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Abstract 

Agent  based  models  have  been  shown  to  be  powerful  tools  in  deseribing 
proeesses  and  systems  eentered  on  individual  behaviors  and  loeal  interactions  (i.e. 
“bottom-up”)  between  specific  entities.  Current  application  areas  tend  to  be  foeused 
within  the  business  and  social  science  arenas,  although  their  usefulness  has  been 
demonstrated  in  the  modeling  of  various  chemistry  and  physics-based  systems. 

Conversely,  many  highly  proeess-oriented  systems,  such  as  manufacturing 
environments,  tend  to  be  modeled  via  “top-down”  methods,  including  discrete  or 
continuous  event  simulations  among  others.  As  a  result,  potentially  critical  attributes  of 
the  entities  or  resourees  modeled  with  these  methods  (spatial  properties,  “learning  eurve” 
or  adaptability)  may  not  be  adequately  captured  or  developed.  This  researeh  develops  an 
agent  based  model  for  applieation  to  a  problem  heretofore  addressed  solely  via  discrete 
event  simulation  or  stochastic  mathematical  models.  Specifically,  a  model  is  construeted 
to  investigate  the  effects  of  differing  levels  of  maintenance  manning  on  sortie  produetion 
eapability,  with  an  examination  of  those  effeets  on  the  resulting  Combat  Mission 
Readiness  (CMR)  of  a  typieal  F-16  squadron. 
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AN  EXPLORATION  OF  THE  EFFECTS  OF  MAINTENANCE  MANNING  ON 
COMBAT  MISSION  READINESS  (CMR)  UTILIZING  AGENT  BASED  MODELING 


1.  Introduction 

1.1  Background 

The  Air  Force  possesses  the  singular  ability  to  “project  national  influence 
anywhere  in  the  world”  on  very  short  notice  (AFDD-1,  2003).  This  is  accomplished  via 
the  application  of  a  broad  spectrum  of  capabilities,  with  the  end  goal  to  be  ever-ready  to 
meet  the  needs  both  of  our  National  leaders  as  well  as  the  combatant  commanders  (Air 
Force  Posture  Statement,  2009).  The  Air  Force’s  primary  force  application  tool  is  the 
aircraft  or  spacecraft,  which  are  operated  and  supported  by  a  host  of  personnel  across  a 
variety  of  organizations. 

Substantial  time  expenditures  in  both  training  and  maintenance  activities  are 
required  to  ensure  the  constant  readiness  of  both  the  operators  and  support  personnel  to 
support  mission  taskings.  As  with  any  complex  organization,  metrics  have  been 
established  for  leaders  to  gauge  progression  and  measure  status  of  processes  and  systems 
critical  to  mission  accomplishment.  A  key  metric  used  by  leadership  to  gauge  the 
service’s  instantaneous  level  of  readiness  to  apply  airpower  is  Combat  Mission  Readiness 
(CMR).  Specifically,  CMR  is  defined  as  “the  minimum  training  required  for  pilots  to  be 
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qualified  and  proficient  in  ail  of  the  primary  missions  tasked  to  their  assigned  unit  and 
weapons  system”  (API  1 1-2F-16V1,  2007). 

An  analysis  was  performed  on  CMR  in  response  to  a  tasking  by  the  commander 
of  United  States  Air  Forces  in  Europe  (USAFE)  (Eipina,  2009).  While  not  an  exact 
measure  of  specific  ability  levels,  the  CMR  metric  provides  leadership  a  top-level  view  of 
a  unit’s  overall  readiness  to  execute  their  assigned  mission  at  any  given  time.  Eipina’s 
research  developed  a  regression  model  to  determine  the  major  factors  driving  CMR.  His 
results  showed  that  CMR  depended  in  large  part  on  availability  of  qualified  aircraft 
maintenance  manpower. 

Currently,  the  active  duty  enlisted  aircraft  maintenance  workforce  makes  up  a 
little  over  27%  of  the  Air  Eorce’s  total  active  enlisted  end-strength  (AE  Personnel  Center, 
2009).  This  figure  encompasses  Airmen  at  all  skill  levels  and  with  Air  Eorce  Specialty 
Codes  (AESCs)  beginning  with: 

•  2A  -  Manned  Aerospace  Maintenance 

•  2P  -  Precision  Measurement  Equipment  Eaboratory 

•  2R  -  Maintenance  Management 

•  2W  -  Munitions  and  Weapons  (Air  Eorce  Enlisted  Classification  Directory,  2009) 
This  27%  figure  can  easily  be  inflated  when  one  considers  the  airmen  involved  in 
delivery  of  fuel,  mission  planning  or  supply  chain  management/parts  delivery;  the  above 
grouping  of  AESCs  is  representative  of  what  would  be  contained  in  a  typical 
Maintenance  Group. 

Directly  related  to  finding  CMR’s  dependence  on  skilled  maintenance  technician 
availability  was  Eipina’s  discovery  of  annual  flying  hour  program  requests  from  a  variety 
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of  bases.  Specifically,  in  fiscal  2009,  fighter  wings  across  USAFE  requested  less  flying 
hours  than  were  actually  required  to  maintain  the  minimum  proficiency  levels  required 
for  each  unit  to  ensure  each  of  their  pilots  remained  CMR.  In  fact,  each  unit  requested 
only  the  number  of  sorties/hours  that  their  respective  Maintenance  Groups  stated  they 
could  produce.  This  fact  supports  Lipina’s  regression  study  results;  a  unit’s  ability  to 
maintain  their  CMR  rates  is  directly  tied  to  maintenance’s  ability  to  provide  sufficient 
mission-ready  aircraft.  This  thesis  explores  this  specific  relationship  by  modeling  the 
effects  of  varied  levels  of  manning  on  a  maintenance  unit’s  sortie  generation  capabilities. 

1.2  Problem  Statement 

There  are  currently  no  models  of  the  sortie  generation  process  that  account  for  the 
adaptive  nature  of  the  extremely  complex  maintenance  process.  Specifically,  no  models 
consider  the  effects  of  various  skill  levels  on  unit  productivity.  This  research  examines 
the  potential  application  of  agent-based  modeling  (ABM)  techniques  to  explore  the 
impacts  and  contributions  of  maintenance  manpower  to  sortie  production  and,  ultimately, 
combat  mission  readiness. 

1.3  Scope 

The  sortie  generation  process  at  a  unit  level  is  a  complex  set  of  processes  with  a 
variety  of  stochastic  elements  and  influences  from  multiple  sources.  Each  of  the 
impactors  and  influences  in  the  overall  process  can  be  considered  a  self-contained  “unit”. 
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Whether  one  contemplates  the  individual  electrician  or  crew  chief  on  the  flight  line,  the 
supply  troop  in  the  Logistics  Readiness  Squadron,  or  even  the  aircraft  being  worked  or 
the  parts  being  moved  up  and  down  the  supply  chain,  each  individual  plays  a 
fundamental  role  in  the  overall  process.  Additionally,  while  the  behaviors  and 
motivations  of  each  of  these  constituent  units  is  relatively  well  understood,  the  resulting 
behavior  of  the  system  as  a  whole  is  more  complex  than  any  explanation  any  individual 
component  could  provide.  This  is  a  hallmark  of  a  complex  system  (Flake,  2002),  an 
environment  well  suited  to  an  ABM’s  ability  to  reveal  “properties  of  systems  that  are  not 
properties  of  the  agents  themselves”  (Jones,  2007). 

1,4  Background  on  ABM 

ABM  is  a  relatively  new  approach  to  the  simulation  of  complex  systems. 

Instances  such  as  Conway’s  1970  “Game  of  Life”  notwithstanding,  the  use  of  agent  based 
modeling  techniques  really  began  to  gain  prominence  in  the  mid  1980’s,  as 
computational  power  became  more  and  more  available.  Typically,  these  methods  have 
been  applied  to  the  social  and  physical  sciences,  encompassing  everything  from  basic 
social  interactions  to  epidemiological  studies  to  detailed  physics-based  models  of  gas 
particle  diffusion.  More  recently,  ABM  modeling  techniques  are  being  applied  to 
problems  such  as  power  distribution  networks  and  economic  markets  that  have 
historically  been  modeled  utilizing  other  more  aggregate  methods  (Macal  &  North, 

2005).  Within  the  Department  of  Defense  (DoD),  ABMs  have  become  widely  used  for 
combat  simulations,  but  in  general,  application  to  the  logistics  domain,  and  specifically 
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military  logistics,  is  still  a  fairly  new  concept  (Tripp  et  al,  2006).  Work  in  the  past 
decade  has  applied  ABMs  to  supply  chain  management  (Krishnamurthy  et  al,  2008),  and 
Tripp  notes  that  these  and  similar  researeh  efforts  are  critical  in  advancing  the  DoD’s 
eapability  to  implement  advanced  eombat  support  proeesses.  The  Marine  Corps 
Warfighting  Laboratory’s  Project  Albert  is  another  signifieant  effort  within  the  DoD  that 
made  use  of  agent  based  models.  While  mueh  of  the  researeh  was  geared  toward  in- 
depth  explorations  of  eombat  operations  and  situations  (Projeet  Albert,  2010),  researeh 
was  performed  on  the  effeet  of  troop  quality  on  eombat  effeetiveness  (Englebaek  et  al, 
2003).  This  is  directly  translatable  to  the  research  and  methodology  deseribed  in  this 
thesis. 

The  sortie  generation  problem  is  not  new.  There  has  been  a  host  of  researeh 
performed  on  the  issue  with  objeetives  spanning  everything  from  general  system 
observation  and  eharaeterization  to  attempts  to  optimize  one  or  more  constituent  sub- 
proeesses  within  the  overall  sortie  generation  proeess.  These  researeh  efforts  have 
employed  many  methods,  ineluding  discrete  event  simulation  (Faas,  2003;  lakovidis, 
2005),  Markov  deeision  analysis  (Dietz,  1991)  and  neural  networks  (Dagg,  1991).  Some 
of  these  have  even  addressed  the  speeifie  issue  of  maintenanee  manning  and  its  potential 
effeets  on  sortie  produetion  and  overall  readiness  (Gotz  and  Stanton,  1986).  However, 
only  a  single  paper  was  loeated  that  dealt  speeifically  with  the  dynamics  of  the  system  as 
influenced  by  maintenanee  skill  levels  (Gareia  and  Raeher,  1981).  Regardless,  the 
methodologies  utilized  to  date  follow  a  more  traditional  approaeh  of  deeomposing  the 
system  under  investigation  and  attempting  to  describe  its  behavior  as  the  “sum  of  its 
parts”,  whieh  has  been  shown  to  be  “inadequate  to  model  and  analyze”  some  large  and 
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complex  systems  (Kaegi  et  al,  2009).  In  fact,  research  performed  across  multiple 
disciplines  has  shown  that  these  traditional  methods  of  system  deeomposition  and 
subsequent  reconstitution  can  prove  not  only  inadequate  but  also  potentially  produce 
misleading  results  (Bobashev  et  al,  2007).  Kaegi  et  al  (2009)  further  argue  that  in  these 
situations,  ABM  has  a  “high  potential  to  help  realistieally  model  large  and  complex 
systems”. 

1,4,1  What  is  an  Agent? 

All  of  the  literature  reviewed  agrees  on  one  thing;  there  is  no  agreed-upon 

definition  of  what  eonstitutes  an  agent.  Franklin  and  Graesser  (1997)  provide  an 

exeellent  overview  of  a  variety  of  definitions  of  what  eonstitutes  an  agent,  and  their 

“Maes  Agent”  is  definitely  germane  to  this  specific  problem; 

Autonomous  agents  are  computational  systems  that  inhabit  some  complex 
dynamic  environment,  sense  and  act  autonomously  in  this  environment,  and  by 
doing  so  realize  a  set  of  goals  or  tasks  for  which  they  are  designed.  (Franklin  and 
Graesser,  1997) 

Another  deseription  labels  agents  as  “autonomous  deeision-making  entities”,  eaeh 
of  which  “individually  assesses  its  situation  and  makes  deeisions  based  on  a  set  of  rules” 
(Bonabeau,  2002).  The  eommon  thread  throughout  the  majority  of  definitions  of  agents 
is  the  idea  of  autonomy,  namely  that  eaeh  of  these  entities  is  not  a  passive  observer  or 
passive  follower  of  directions,  but  rather  an  active  unit  executing  its  own  set  of  behaviors. 

An  additional  eritieal  property  of  agents  is  their  collective  ability  to  exhibit 
eomplex  patterns  of  behavior  that  might  not  have  been  readily  evident  when 
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contemplating  the  properties  of  the  individual  agents  themselves.  These  so-ealled 
“emergent  behaviors”  (Gilbert  and  Terna,  1999)  are  a  produet  not  only  of  the  behaviors 
of  eaeh  of  the  individual  agents,  but  also  of  the  means  by  whieh  the  agents  internet  and 
the  level  of  interaction  stemming  from  their  individual  behaviors. 

All  models  are  an  abstraetion  of  a  system  utilizing  some  means  of  aggregation  to 
deseribe  system  behaviors.  ABM’s  focus  on  the  behaviors  of  individual  entities,  or 
“agents”,  allow  it  to  eapture  interaetions  between  distinet  individuals  or  groups.  In  the 
eontext  of  the  sortie  generation  proeess,  this  ineludes,  among  other  things,  the  skill  levels 
and  related  effieieney  of  individual  maintainers  and  the  ehange  in  system  response  as 
eaeh  individual  “learns”  how  to  better  execute  their  job.  While  this  may  be  a  relatively 
simplistie  example,  the  more  traditional  modeling  teehniques  utilized  in  the  past  fail  to 
eapture  this  level  of  system  detail,  instead  relying  on  either  static  values  or  “one-size-fits- 
all”-litted  theoretieal  distributions  aimed  at  providing  the  aggregated  swath  of  “most 
likely”  values. 

Exploring  the  sortie  generation  and  maintenanee  manning  problems  with  ABM 
teehniques  provides  a  theoretical  counterpoint  to  the  previous  body  of  work.  In  applying 
these  new  teehniques,  new  insight  ean  be  gained  into  both  the  internal  system  behaviors 
and  resulting  system  responses.  Additionally,  a  sueeessful  modeling  effort  will  indieate 
that  these  teehniques  are  not  only  applieable  to  this  body  of  problems,  but  might  also  be 
preferable,  both  due  to  its  ability  to  eapture  emergent  phenomena  and  its  touted  flexibility 
as  a  modeling  domain  (Bonabeau,  2002). 
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1,5  Why  Sortie  Generation? 


Despite  a  multitude  of  mission  areas  eovering  a  broad  speetrum  of  domains,  the 
Air  Foree  remains  a  predominantly  aerospaee-foeused  serviee.  Specific  to  this  research, 
an  individual  unit’s  CMR  rates  are  direct  results  of  the  number  of  Ready  Aircrew 
Program  (RAP)-coded  sorties  successfully  completed  by  aircrews  each  month  (AFIl  1- 
2F-16  VI,  2007).  A  unit’s  ability  to  successfully  generate  sorties  holds  a  prominent 
position  among  the  key  factors  affecting  RAP  completion  and,  subsequently,  CMR. 

Figure  1  provides  a  top-level  view  of  the  sortie  generation  flow.  The  process  is 
typically  considered  to  start  with  the  recovery  of  an  aircraft  from  a  previous  mission 
(AFI21-101,  2006).  After  completion  of  recovery  actions  and  placing  the  aircraft  on  a 
parking  spot,  the  aircraft  is  serviced  and  the  aircrew  provides  details  of  any  issues 
encountered  during  flight.  Depending  on  the  severity,  or  if  items  are  discovered  during 
servicing  operations,  unscheduled  maintenance  actions  are  initiated.  Depending  on 
necessity  and  resource  availability  (time,  parts,  technicians,  etc.)  preventative 
maintenance  measures  may  also  be  started  at  this  point.  While  schedules  are  typically  set 
at  least  a  week  ahead  of  time,  some  online  changes  may  need  to  be  made  at  this  point  due 
to  aircraft  status  changes.  The  aircraft  is  reconfigured  for  its  next  mission,  receives  a  set 
of  inspections,  and  is  then  launched  on  its  next  sortie. 


Figure  1  -  Sortie  Generation  Process  (Faas,  2003) 


Each  of  the  sub-process  blocks  in  Figure  1  is  a  self-contained  complex  procedure 
and  depends  on  a  variety  of  internal  (various  flight-line  maintenance  specialties)  and 
external  (supply,  fuel  delivery,  etc.)  stakeholders.  Not  all  of  these  sub-processes  are 
explicitly  modeled  in  this  research. 

1,6  Methodology 

The  current  model  was  developed  around  the  sortie  generation  process  and 
centers  on  activities  performed  by  a  typical  Aircraft  Maintenance  Squadron.  This  aligns 
with  the  research’s  primary  focus  on  a  unit’s  operational  readiness  as  a  function  of 
maintenance  manning  levels.  The  focus  is  not  on  preventive  or  off-equipment 
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maintenance  tasks  even  though  such  efforts  do  support  the  overall  goal  of  operational  and 
combat  readiness.  However,  with  many  of  the  measures  centered  around  sortie 
production  rates  and  the  activities  related  to  this  production,  it  was  natural  to  scope  the 
research  to  a  single  area  within  a  Maintenance  Group.  This  scoping  is  not  unusual 
(Lipina,  2009;  Chimka  and  Nachtmann,  2007).  This  research  also  extends  previous  work 
in  the  same  area  (Lipina,  2009),  which  focused  on  a  single  unit.  Due  to  Lipina’s  focus, 
and  work  with  Spangdahlem  AB,  the  same  flying  unit  was  selected,  both  to  serve  as  a 
point  of  comparison,  and  since  a  multitude  of  data  was  readily  available. 

The  body  of  previous  work  revealed  very  few  analyses  of  the  effects  of 
maintenance  technician  expertise  on  sortie  production.  Chimka  and  Nachtman  (2007) 
provide  a  regression  study  on  which  manning  factors  were  used  to  forecast  fleet  health 
metrics.  Garcia  and  Racher  (1981)  examined  the  influences  of  skill  levels  within  a 
maintenance  setting.  Their  work  focused  on  a  single  set  of  Air  Force  Specialty  Codes 
(AFSCs)  and  a  methodology  to  modify  the  Logistics  Composite  Model  (LCOM)  to 
account  for  differences  in  skill  sets.  They  concluded  that  the  skill  levels  of  maintainers 
significantly  affected  the  productivity  of  a  maintenance  workcenter.  This  supports  the 
hypothesis  that  an  analysis  of  a  maintenance  workforce  should  include  the  effects  due  to 
skill  level  mix. 

LCOM  is  a  “statistical  simulation  model  that  the  Air  Force  uses  to  estimate 
monthly  man-hours  and  shift  manning  required  to  accomplish  direct  maintenance  tasks” 
(Dahlman  et  al,  2002).  Simply  put,  it  is  a  non-agent-based  simulation  tool  used  by  the 
Air  Force  to  determine  the  appropriate  manning  for  a  maintenance  unit,  based  on  a 
detailed  simulation  of  aircraft  and  supporting  maintenance  operations  at  the  unit  level. 
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While  the  eapability  exists  to  implement  “work  arounds”  to  model  the  effeets  of  various 
skill  levels  on  fix  times,  ete.,  this  requires  the  user  to  modify  the  detailed  task  network 
definitions  within  LCOM.  These  modifications  establish  lower  skill  levels  as  allowable 
substitutions  for  tasks,  and  then  apply  task  multipliers  when  those  substitutions  are  made. 
Besides  being  cumbersome  to  employ,  these  approaches  fail  to  effectively  capture  the 
resources  being  applied  for  on  the  job  training  (OJT)  during  task  execution,  and  also  fail 
to  depict  the  ability  for  technicians  to  “learn”  and  improve  in  efficiency  over  time.  The 
fact  remains,  LCOM  “does  not  explicitly  account  for  OJT  or  experience  mix”  (Dahlman 
et  al,  2002),  and  the  emergent  behavior  of  the  overall  system  as  each  maintainer  improves 
in  efficiency  and  productivity  over  time  make  this  an  excellent  problem  to  be  tackled 
with  an  ABM. 

1.7  Model  Construction 

Law  (2007)  provides  a  framework  for  designing  a  simulation  study;  his  method 
was  used  as  a  guide.  Figure  2  details  the  study  design  flow. 
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Figure  2  -  Steps  in  a  Simulation  Study  (Law,  2007) 


The  modeling  proeess  begins  with  a  general  layout  of  the  logic  and  process  flows 
involved.  This  macro-level  flow  is  developed  based  on  personal  experience,  with  some 
details  on  idiosyncrasies  of  fighter  aircraft  maintenance  procedures  provided  by 
interviews  with  AFIT  graduate  students  with  background  in  maintenance  of  fighter 
aircraft,  and  maintenance  technicians  in  the  field.  The  overall  model  flow  is  a  sub-set  of 
the  top-level  sortie  generation  process  depicted  in  Figure  1,  and  is  included  in  Figure  3. 
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Figure  3  -  Modeled  Sortie  Generation  Flow 


Figure  3  shows  that  the  modeled  sortie  generation  flow  includes  all  but  two  of  the 
top  level  flow  process  blocks  as  proposed  by  Faas  (2003)  (see  Figure  1).  Additionally, 
the  “Aircraft  Scheduling”  block  is  moved  to  the  center  of  the  graph.  This  represents  the 
scheduling  of  individual  aircraft  as  a  continuously  executed  task. 

This  model  was  developed  on  a  “ground-up”  framework,  using  an  agent-based 
modeling  environment  called  NetLogo,  developed  at  Northwestern  University  (Wilensky, 
1999).  The  development  environment  was  selected  based  on  ease  of  use  and  the  system’s 
operating  characteristics  (Railsback  et  ah,  2006).  One  chief  concern  was  the  robustness 
of  the  random  number  generator,  and  the  ability  of  the  system  to  designate  separate  and 
distinct  random  number  seeds  to  specific  processes  in  order  to  implement  common 
random  numbers  as  a  variance  reduction  technique  during  the  analysis  period.  The 
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NetLogo  system  did  not  have  the  eapability  to  traek  more  than  a  single  random  number 
seed  at  a  time.  A  JAVA-based  extension  was  developed  to  add  this  eapability  (all  source 
code  and  executables  are  available  upon  coordination  with  AFIT).  The  random  number 
generator  used  in  the  NetLogo  environment  is  the  Mersenne  twister,  proposed  by 
Matsumoto  and  Nishimura  (1998).  An  in-depth  statistical  evaluation  is  performed  in 
(L’Ecuyer,  2001)  and  the  generator  was  found  to  perform  as  well  as  generators  used  in  a 
variety  of  commercial  simulation  software  packages. 

To  develop  the  model  framework,  the  overarching  process  flows  were  abandoned 
in  favor  of  “behavior”  models  for  each  of  the  identified  agents.  In  this  model,  there  were 
4  identified  Agent  types.  These  were: 

•  Maintainers  -  Each  maintainer  was  defined  based  on  their  AESC,  to 
include: 

o  Crew  Chief  (CC), 
o  Avionics, 

o  Electro-Environmental  (EE),  and 
o  Propulsion  (Prop). 

Each  of  these  were  then  further  classified  by  skill  level  (3-level  Apprentices,  5- 
level  Journeymen  and  7-level  Craftsmen  ).  AEI  36-2101  (2009)  provides  key 
characteristics  for  each  skill  level: 

3-level:  “basic  knowledge  in  their  AESC  through  completion  of  initial  skills 
training”, 

5-level:  “demonstrated  skilled  proficiency  in  their  AESC”,  and 
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7-level:  “high  degree  of  teehnieal  knowledge  within  their  AFSC  and. .  .have 
aequired  supervisory  eapability  through  training  and  experienee”. 

•  Production  Superintendent  (Pro  Super)  -  The  pro  super  direets  the  unit’s 
overall  maintenanee  effort. 

•  Expediters  -  Expediters  work  direetly  for  the  pro  super  and  ensure 
maintenanee  is  aeeomplished,  managing  their  assigned  pools  of  resourees 
(personnel)  to  meet  the  pro  super’s  direetion.  The  three  sub-types  are: 

o  Crew  Chief  Expediter 
o  Meehanieal  Expediter  (EE,  Prop,  Hydro) 
o  Avionies  (Comm/Nav,  GAC) 

•  Aircraft  -  Aircraft  agents  within  the  model  are  simple  entities  with  no  real 
behaviors  of  note.  These  entities  serve  to  provide  tangible  targets  for  the 
maintenance  agents,  as  well  as  holding  a  variety  of  variables  that  support  the 
overall  logic  flow  of  the  model. 

Chapter  2  provides  details  on  the  development  of  the  model  and  various  analytical 
results.  Chapter  3  presents  an  application  of  the  model  to  a  representative  case  study 
focused  on  a  single  “typical”  aircraft  maintenance  unit,  along  with  numerical  results. 
Chapter  4  concludes  the  thesis,  highlighting  significant  findings  and  identifying  areas  for 
future  study.  Note  Chapters  2  and  3  are  structured  as  standalone  papers. 
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2.  Application  of  Agent  Based  Modeling  to  the  Sortie 

Generation  Problem 


2,1  Overview 

Substantial  time  expenditures  for  both  training  and  maintenanee  aetivities  are 
required  to  ensure  the  eonstant  readiness  of  operators  and  support  personnel  to  support 
mission  taskings  within  the  United  States  Air  Foree.  As  with  any  eomplex  organization, 
metries  have  been  established  for  leaders  to  gauge  progression  and  measure  status  of 
proeesses  and  systems  eritieal  to  mission  aeeomplishment.  A  key  metrie  used  by 
leadership  to  gauge  the  serviee’s  instantaneous  level  of  readiness  to  apply  airpower  is 
Combat  Mission  Readiness  (CMR).  Speeifieally,  CMR  is  defined  as  “the  minimum 
training  required  for  pilots  to  be  qualified  and  profieient  in  all  of  the  primary  missions 
tasked  to  their  assigned  unit  and  weapons  system”  (AFI  1 1-2F-16V1,  2007). 

An  analysis  was  performed  on  CMR  in  response  to  a  tasking  by  the  eommander 
of  United  States  Air  Forees  in  Europe  (USAFE)  (Eipina,  2009).  While  not  an  exaet 
measure  of  speeifie  ability  levels,  the  CMR  metrie  provides  leadership  a  top-level  view  of 
a  unit’s  overall  readiness  to  exeeute  their  assigned  mission  at  any  given  time.  Eipina’s 
researeh  developed  a  regression  model  to  determine  the  major  faetors  driving  CMR.  His 
results  showed  that  CMR  depended  in  large  part  on  availability  of  qualified  aireraft 
maintenanee  manpower. 

This  researeh  develops  an  agent-based  simulation  model  for  applieation  to  the 
sortie  generation  proeess,  foeused  on  an  individual  unit.  The  simulation  ineludes 
representations  of  eaeh  individual  maintainer  within  the  unit,  along  with  supervisory 
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agents  that  provide  direetion  in  the  form  of  task  prioritization  and  resource  assignment. 
Using  a  high-fidelity  depiction  of  each  entity,  an  exploration  of  the  effects  of  different 
mixes  of  skill  levels  and  Air  Force  Specialty  Codes  (AFSCs)  on  sortie  production  is 
performed.  The  model  development  was  executed  in  three  distinct  phases: 

Agent  definition, 

Supporting  data  extraction  and  filtering,  and 

Model  coding  and  supporting  logic  development. 

The  sortie  generation  problem  is  not  new.  There  has  been  a  host  of  research 
performed  on  the  issue  with  objectives  spanning  everything  from  general  system 
observation  and  characterization  to  attempts  to  optimize  one  or  more  constituent  sub¬ 
processes  within  the  overall  sortie  generation  process.  These  research  efforts  have 
employed  many  methods,  including  discrete  event  simulation  (Faas,  2003;  lakovidis, 
2005),  Markov  decision  analysis  (Dietz,  1991)  and  neural  networks  (Dagg,  1991).  Some 
of  these  efforts  have  even  addressed  the  specific  issue  of  maintenance  manning  and  its 
potential  effects  on  sortie  production  and  overall  readiness  (Gotz  and  Stanton,  1986). 
Regardless,  the  methodologies  utilized  follow  a  more  traditional  approach  of 
decomposing  the  system  under  investigation  and  attempting  to  describe  its  behavior  as 
the  “sum  of  its  parts”,  which  has  been  shown  to  be  “inadequate  to  model  and  analyze” 
some  large  and  complex  systems  (Kaegi  et  al,  2009).  In  fact,  research  performed  across 
multiple  disciplines  has  shown  that  these  traditional  methods  of  system  decomposition 
and  subsequent  reconstitution  can  prove  not  only  inadequate  but  also  can  potentially 
produce  misleading  results  (Bobashev  et  al,  2007).  Kaegi  et  al  (2009)  further  argue  that 
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in  these  situations,  ABM  has  a  “high  potential  to  help  realistically  model  large  and 
complex  systems”. 

2.2  Agent  Development 

The  power  of  the  agent-based  model  environment  lies  in  its  ability  to  codify  a  specific 
agent’s  decision-making  processes  into  behaviors  (often  as  simple  rule-sets)  and  then 
observe  the  agents  as  they  “autonomously”  react  and  interact  with  other  agents  and  their 
environment,  potentially  collectively  producing  “emergent  behaviors”  that  might  not 
otherwise  have  been  either  observed  or  predicted  based  on  other  methods  of  analysis 
(Bonabeau,  2002).  In  the  case  of  the  current  model,  specific  logic  flows  were  mapped  for 
each  agent  type,  similar  to  the  process  used  when  developing  the  general  flow.  Each 
flow  was  developed  using  the  agent’s  point  of  view  and  level  of  global  “awareness”, 
which  varied  depending  on  the  agent  type.  In  the  interest  of  standardization  and 
readibility  of  the  agent  descriptions,  details  on  each  of  the  agent  types  are  provided  below 
in  accordance  with  applicable  sections  of  the  Overview,  Design  Concepts  and  Details 
(ODD)  protocol  (Grimm  et  al.  2006). 

2.2,1  Aircraft  Agents 

Each  aircraft  agent  is  a  purely  reactive  entity.  It  contains  attributes  that  track 
completion  of  prefiights,  sortie  counts,  and  current  status  among  other  things,  but  does 
not  employ  any  active  decision-making  functions.  Based  on  data  obtained  from 
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Spangdahlem  AB,  22  aircraft  were  incorporated  into  the  model.  Break,  abort  and  other 
data  measuring  failures  of  the  aircraft  were  gathered  and  utilized  to  construct  empirical 
and/or  theoretical  distributions  for  use  in  describing  the  stochastic  nature  of  the  various 
failure  mechanisms  affecting  each  aircraft. 

The  aircraft  agent  has  two  system  states:  non  mission  capable  (NMC)  and  mission 
capable  (FMC).  No  partially  mission  capable  (PMC)  state  is  included  (Ciarallo  et  al 
(2005)  do  model  PMC  states  for  mobility  aircraft).  This  was  done  to  simplify  the  overall 
state  space  of  the  model,  and  since  the  other  logic  mechanisms  do  not  require  the  addition 
of  a  third  state.  Figure  4  provides  the  modeled  flows  for  aircraft  agents. 


Figure  4  -  Aircraft  Agent  Logic 
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2,2.2  Production  Supervisor  Agent 


As  defined  in  API  21-101  (2006),  the  production  supervisor  (Pro  Super)  is 
responsible  for  directing  “the  overall  maintenance  effort  of  their  unit.”  As  such,  the  Pro 
Super  agent  is  the  only  one  with  true  global  awareness,  and  makes  the  majority  of 
decisions  within  the  simulation.  These  include: 

Job  priorities  as  they  arise. 

Which  aircraft  are  to  be  put  into  the  flying  schedule. 

How  many  and  which  aircraft  are  to  be  generated  as  spares,  and 
When  to  begin  work  on  what  aircraft. 

The  Pro  Super  has  two  states:  available/planning  and  exceptional  release  (ER)  signoff. 
In  the  former,  he  is  performing  each  of  the  tasks  outlined  above  every  time  step.  For  the 
latter,  he  is  considered  unavailable  while  signing  the  ER.  As  the  individual  with  overall 
responsibility  for  maintenance  execution,  the  Pro  Supers’  signing  of  the  ER  “serves  as 
certification... that  the  [aircraft]  is  safe  for  flighf’  (TO  00-20-1,  2006)  and  is  required 
prior  to  each  sortie.  Figure  5  depicts  the  Pro  Super  agent  logic. 
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Figure  5  -  Pro  Super  Ageut  Logic 


2,2,3  Expediter  Agents 

Each  of  the  three  expediter  agents  have  a  redueed  level  of  global  awareness.  The 
crew  ehief  expediter  is  speeifieally  aware  of  any  impending  aircraft  landings,  since 
aircraft  tend  to  be  reeovered  by  erew  ehief  personnel.  Expediters  also  perform  the 
alloeation  of  maintenance  teehnieians  to  tasks  based  on  the  task’s  priority  as  assigned  by 
the  produetion  superintendent  agent.  By  definition,  expediters  “work  for  the  Pro  Super 
and  manage,  eontrol  and  direet  resourees”  in  order  to  “ensure  maintenanee  is 
aceomplished”  (API  21-101,  2006).  Within  their  exeeution  time  step,  eaeh  of  the 
expediters  sean  for  jobs  in  the  system  (taskings  from  the  produetion  superintendent),  and 
if  any  are  found  for  the  AESCs  they  are  responsible  for  (note  above  that  eaeh  expediter  is 
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responsible  for  a  specifie  subset  of  the  overall  mix  of  maintenance  AFSCs)  they  proceed 
with  their  job  assignment  logic. 

For  expediters  to  assign  a  task  to  a  group  of  maintainers,  they  must  first  determine 
if  sufficient  manpower  is  currently  available.  One  key  consideration  is  the  number  of 
maintainers  required  for  each  job.  In  many  cases,  specific  tasks  carry  technical  order 
requirements  for  minimum  crew  sizes.  While  the  current  model  does  not  include 
sufficient  detail  to  capture  the  actual  task-level  crew-size  requirements,  this  influence  is 
captured  by  treating  crew  size  as  a  random  variable  and  drawing  from  an  empirical 
distribution  based  on  two  years  worth  of  data  from  Spangdahlem  AB.  This  determines 
crew  size  based  on  the  work  unit  code  (WUC)  of  the  job  and  the  AFSC  assigned  to  work 
the  job.  Depending  on  the  priority  of  the  task  and  the  availability  of  personnel  working 
lower  priority  jobs,  the  expediter  may: 

Pull  personnel  from  lower  priority  jobs. 

Delay  the  job,  or 

Work  the  job  with  the  available  (sub-optimal)  manpower  with  a  penalty  on  job 
time. 

Additionally,  when  assigning  a  job,  expediters  must  also  determine  when  to  allow 
training  to  occur,  and  whether  or  not  training  (when  allowed)  will  occur.  This  is 
determined  based  on  the  priority  of  the  job  (priority  one  jobs  do  not  allow  for  training) 
and  the  skill  level  of  the  initially  assigned  team.  Once  a  fully  qualified  (5  and  7  levels) 
team  has  been  selected  by  the  expediter,  a  random  draw  is  evaluated  against  the  lowest 
efficiency  value  on  the  qualified  team  (see  section  below  for  efficiency  value).  If  the 
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random  number  is  lower,  then  training  is  permitted  to  oecur  and  up  to  two  3 -levels  are 
randomly  selected  to  be  trained.  The  Expediter  agent  logic  is  included  below  in  Figure  6. 


Figure  6  -  Expediter  Agent  Logic 


2,2,4  Maintainer  Agents 

The  individual  maintainer  agents  have  a  minimum  of  global  awareness.  Given  that 
the  typical  maintainer’ s  focus  is  on  fixing,  inspecting,  or  servicing  an  aircraft,  there  was 
no  need  to  model  any  decision-making  capabilities.  In  effect,  each  maintainer  resides  in 
a  ready  pool  until  tasked  to  a  job  by  their  owning  expediter.  Logic  to  determine  the 
nature,  fix- time,  crew  size,  etc.  of  the  tasks  are  driven  by  random  draws  evaluated  against 
empirical  and/or  theoretical  distributions  derived  from  data  gleaned  from  the  CAMS 
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maintenance  information  system  (MIS).  When  assigned  to  a  job,  each  individual’s 
efficiency  attribute  is  used  to  determine  the  speed  at  which  it  is  accomplished.  An 
individual  agent’s  efficiency  value  attribute  varies  from  0  (no  skills)  to  1  (highly  skilled). 
This  attribute  is  then  increased  over  time,  via  a  learning  curve  function  developed 
through  interviews  with  experienced  maintainers  both  in  the  field  and  at  AFIT.  General 
logic  flow  is  depicted  in  Figure  7. 


Figure  7  -  Maintenance  (mx)  Agent  Logic 
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2,3  Supporting  Data 


The  sortie  generation  process  is  data  intensive,  even  when  examined  at  a 
aggregate  or  very  top  level.  As  increasing  levels  of  detail  were  added  to  each  of  the 
agents,  requirements  for  additional  data  increased.  Primary  sources  included  the 
Logistics  Installations  and  Mission  Support  Enterprise  View  (LIMS-EV),  and  the  Global 
Combat  Support  System  -  AE  Data  Services  (GCSS-AE),  both  web-based  tools 
accessible  via  the  Air  Eorce  Portal.  Additional  data  was  provided  by  USAPE/A9, 
ACC/A4  and  interviews  with  maintainers  at  both  Shaw  APB  and  Spangdahlem  AB. 
Table  1  lists  key  data  requirements  and  their  sources. 


Table  1  -  Data  Requirements  and  Sourcing 


Data  Reauirement 

Source 

#  of  Aircraft 

Spangdahlem  AB 

#  of  Personnel 

Shaw  APB 

Break  Rate 

EIMS-EV,  GCSS 

Abort  Rate 

LIMS-EV,  GCSS 

Pix  Rates 

GCSS 

Work  Unit  Code  (WUC)  Determination 

GCSS 

APSC  Assignment 

GCSS 

Crew  Size  Determination 

GCSS 

Average  Sortie  Duration  (ASD) 

LIMS-EV 

Eearning  curve 

Shaw  APB 

25 


A  large  number  of  diserepaneies  were  noted  during  eolleetion  of  the  maintenanee 
data  from  GCSS.  Speoifieally,  after  eolleeting  data  on  unseheduled  maintenanee 
performed  over  a  two-year  time  period,  almost  30%  of  the  items  attributed  to  one  AFSC 
were  aetually  scheduled  inspections  that  appeared  to  have  been  incorrectly  coded  in  the 
system  as  unscheduled  maintenance  tasks.  Potential  impacts  due  to  constraints  on  the 
accuracy  of  data  within  automated  maintenance  information  systems  have  been  discussed 
in  other  studies  (Dahlman  et  al,  2002).  Efforts  were  made  to  filter  out  inconsistencies 
that  were  readily  apparent,  but  a  key  assumption  is  that  the  remaining  data  used  in  our 
study  is  representative  of  true  system  behavior  and  performance. 

While  in  some  instances  the  raw  data  could  be  used  directly  (numbers  of  aircraft 
and  personnel),  in  the  majority  of  cases  distributions  had  to  either  be  constructed  or  fitto 
the  gathered  data.  This  was  further  delineated  into  those  instances  requiring  the 
construction  of  simple  distributions  and  those  requiring  the  formulation  of  a  more 
complex  conditional  structure.  Distributions  for  break  and  abort  rates  are  examples  of  the 
simpler  case,  where  each  was  represented  with  a  single,  simple,  theoretical  distribution  or 
empirical  distribution  function.  Conversely,  WUC  determination  and  fix  rates  provide  an 
example  of  the  more  complex  structure.  Rather  than  utilizing  a  single  distribution  to 
characterize  a  general  fix  time  for  each  break,  the  model  determines  a  WUC  for  each 
break,  a  specific  AFSC  set  to  work  that  WUC,  and  then  utilizes  a  specifically  fitted 
theoretical  distribution  based  on  these  conditions  to  calculate  a  fix  time.  This  multi-tiered 
approach  provides  a  more  realistic  portrayal  of  the  process,  both  in  terms  of  a  more 
accurate  depiction  of  manpower  allocation  (specific  AFSCs)  and  a  fix  time  associated 
with  that  allocation. 
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Figure  8  provides  a  graphical  depiction  of  this  WUC  and  AFSC  determination 
process.  Entering  from  the  left  with  a  break  event,  the  model  first  determines  the  WUC 
for  the  aircraft  break  (region  1).  This  is  accomplished  via  an  empirical  distribution 
function  (EDF)  built  from  a  two-year  sample  of  data.  This  data  is  also  situationally 
dependent;  the  EDF  for  breaks  determined  in  flight  (air  aborts  or  Code  3  breaks)  is 
different  from  the  distribution  used  for  breaks  determined  during  an  inspection  or  other 
on-ground  maintenance. 
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The  next  step  is  determining  the  AFSC  required  to  fix  this  specific  break  (region  2 
in  Figure  8).  This  also  uses  an  EDF,  conditional  on  the  pre-determined  WUC.  The  third 
and  final  step  is  the  determination  of  the  base  fix  time  (region  3).  Each  endpoint  of  this 
tree  structure,  representing  a  specific  AESC  and  WEIC  combination,  has  its  own  fitted 
theoretical  distribution.  The  model  performs  a  draw  from  the  specified  distribution,  is 
potentially  modified  depending  on  the  assigned  crew’s  skill  level  composition  and  size, 
and  is  returned  as  a  final  fix  time.  A  similar  multi-tiered  conditional  structure  is  used  for 
determination  of  crew  sizes  for  each  job.  In  each  case,  distributions  were  based  on  two 
years  of  source  data. 

While  the  data-gathering  process  was  by  necessity  a  manual  operation,  fitting  the 
data  to  appropriate  distributions  and  placing  these  into  a  format  usable  by  the  model  was 
automated  to  as  large  a  degree  as  possible.  Ehilizing  routines  developed  in  Microsoft 
Visual  Basic  for  Applications  (VBA)  from  within  Microsoft  Excel,  distributions  were 
calculated  and  then  exported  into  text  files  in  formats  used  by  the  simulation  model. 

EDEs  were  calculated  wholly  within  the  Excel  application,  while  Rockwell  Automation 
Technology’s  Input  Analyzer  software  was  used  to  determine  theoretical  distribution 
parameters  and  return  these  to  Excel  for  subsequent  formatting  and  exporting  operations. 

The  only  exception  to  using  historical  data  and  the  fitting  process  described  above 
was  the  development  of  the  learning  curve  parameters.  This  data  was  obtained  from 
interviews  with  senior  maintenance  personnel  at  Shaw  AEB,  and  consisted  of  estimates 
of  worker  efficiency  by  both  AESC  and  skill  level,  along  with  an  estimate  of  how  long  it 
would  take  to  improve  that  efficiency  to  a  level  on  par  with  the  next  highest  skill  level. 
Utilizing  manpower  availability  estimates  published  in  AEI  38-201  (2003),  these  upgrade 
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times  and  efficiency  figures  were  used  to  form  linear  plots.  The  slope  of  each  plot  was 
then  taken  as  the  efficiency  improvement  “learning  curve”  and  used  to  calculate  an 
efficiency  improvement  at  the  completion  of  every  maintenance  task.  This  work  did  not 
consider  loss  of  learning  due  to  missed  training  or  lack  of  task  accomplishments. 

Figure  9  depicts  the  learning  curves  calculated  for  the  Avionics  AFSC. 


Figure  9  -  Avionics  Learning  Curves 


29 


2,4  Logic  Development 


The  final  step  in  the  model  development  was  the  design  and  implementation  of  an 
overarching  construct  that  established  the  agents,  defined  their  environment  and 
behaviors,  and  provided  a  backdrop  to  track  their  iterative  execution.  Graphical 
depictions  of  each  agent  type’s  behavior  map  and  accompanying  pseudocode  were  used 
extensively  which,  given  the  simulation’s  largely  modular  architecture,  significantly 
aided  the  development  effort. 

The  remainder  of  this  effort  dealt  with  the  addition  of  supporting  logic  and 
routines  to  manage  processes  such  as  shift  change,  input  and  output  of  data,  and  the 
establishment  of  a  standard  work  week  and  flying  schedule.  As  the  model  took  shape, 
multiple  assumptions  had  to  be  made  in  order  to  effectively  scope  the  development  effort. 
Significant  assumptions  and  notes  on  each  are  listed  below: 

Flying  window  remains  constant 

o  While  the  actual  schedule  changes  from  day-to-day,  the  general  flying 
window  remains  on  a  day  shift. 

Flying  load  retains  a  set  schedule 

o  The  sortie  load  varies  from  day-to-day  but  retains  a  set  pattern  that 
repeats  from  week-to-week. 

Aircraft  configuration  is  not  a  concern 

o  Reconfiguration  of  aircraft  for  different  missions  was  not  modeled. 
Based  on  interviews  with  maintenance  production  personnel,  it  was 
determined  that  strict  management  of  aircraft  could  avoid  the  majority 
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of  impacts  due  to  aircraft  reconfiguration.  For  purposes  of  the 
simulation,  it  is  assumed  that  aircraft  with  the  correct  configuration  are 
assigned  to  their  respective  matching  mission  types. 

Scheduled  maintenance  is  not  modeled 

o  Since  phase  inspections  are  performed  outside  of  the  AMU,  this 
process  is  not  modeled.  Other  scheduled  maintenance  actions  are  not 
included  due  both  to  the  great  number  of  scheduled  inspections  and 
maintenance  activities  and  the  lack  of  specific  fix  time  data  for  each. 
Overtime  is  permitted  (up  to  30  minutes)  if  agents  are  working  at  shift  change 
o  Rather  than  attempting  to  hand  over  a  job  near  completion,  if  an 
agent’s  estimated  time  to  completion  (ETIC)  is  within  30  minutes  at 
shift  change,  they  continue  to  work  the  job  until  finished 
Aborts  and  Code  3  breaks  result  in  automatic  cancellations  of  any  hot-pit 
sorties  directly  following 

o  Since  aircraft  landing  Code  3  or  Aborting  are  landing  in  a  non- 
mission-capable  (NMC)  status,  this  forces  the  cancellation  of  any 
follow-on,  hot-pit  sortie  in  order  to  allow  maintenance  to  work  the 
issue. 
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2,5  Experimental  Design  and  Methodology 


The  key  focus  for  the  analysis  with  this  model  was  the  effects  of  varied  levels  of 
maintenance  manpower,  both  in  terms  of  sortie  production  and  utilization.  Specific 
responses  of  interest  were; 

Daily  utilization  rates  for  each  AFSC  and  skill  level,  and 
Sorties  flown  and  cancelled  per  week 

Factors  identified  for  analysis  were  the  number  of  personnel  within  each  AFSC.  While  it 
is  acknowledged  that  different  strategies  for  allocating  personnel  across  shifts  generate 
significantly  different  results,  the  vast  number  of  potential  combinations  and  approaches 
suggested  limiting  design  complexity.  For  this  research,  four  aggregated  manning  factors 
were  utilized,  one  for  each  AFSC.  Each  of  these  had  three  levels: 

Base:  the  typical  manning  profile  based  on  Shaw  AFB’s  daily  manning. 
Reduced:  a  10%  loss  of  personnel,  and 
Increased;  a  10%  increase  of  personnel. 

Rather  than  turn  this  analysis  into  an  exploration  for  an  optimum  AFSC  and  skill 
mix  for  each  shift  in  the  face  of  changing  manning  availability,  both  the  reduced  and 
increased  cases  were  calculated  in  as  straightforward  a  manner  as  possible.  Under  the 
assumption  that  a  unit  would  maintain  relatively  the  same  proportions  of  skills  and 
AFSCs  on  each  shift  regardless  of  manning  changes,  each  shift  and  AFSC  is  evaluated 
individually.  As  an  example,  on  day  shift  there  are  30  crew  chiefs  in  the  base  case:  7 
three-levels,  14  five-levels  and  9  seven-levels.  The  reduced  case  then  translates  to  a  loss 
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of  three  erew  ehiefs  on  this  shift.  Using  the  original  proportions  of  skill  levels  as  a 
reference  (23%,  46%  and  30%  for  three-,  five-  and  seven-levels  respectively),  these 
proportions  are  applied  to  the  reduced  shift  manning  level  of  27  crew  chiefs.  In  this 
example,  the  reduced  levels  of  manning  then  equate  to  6  three-levels,  13  five-levels  and  8 
seven-levels. 

Despite  this  clear-cut  approach  at  testing,  the  running  of  a  complete  full  3"^ 
factorial  experiment  was  estimated  to  take  over  fifteen  days.  However,  since  cross¬ 
utilization  training  was  not  modeled  (there  is  no  sharing  of  job  taskings  between  AFSCs), 
it  was  believed  that  none  of  the  identified  responses  would  exhibit  significant  interactions 
between  any  of  the  experimental  factors.  To  reduce  the  computational  burden  but  still 
obtain  information  on  the  factors  of  interest,  a  fractional  factorial  screening  experiment 
was  employed.  Additionally,  the  factor  space  was  reduced  to  include  only  the  reduced 
and  increased  manning  levels,  with  the  base  levels  used  as  center  points.  As  suggested  in 
Montgomery  (2009),  this  provides  a  means  to  identify  curvature  (quadratic  effects)  and 
test  for  lack  of  fit,  while  simultaneously  minimizing  the  size  and  design  complexity  of  the 
overall  experiment.  This  led  to  the  implementation  of  a  resolution  IV,  2"^'^  design  with  25 
centerpoints  used  as  a  screening  experiment. 

For  the  execution  of  the  experiment,  it  was  determined  that  the  simulation  would 
run  for  a  total  of  210  days.  This  equated  to  7.5  “months”  of  28  calendar  days  (20 
working  days)  each.  This  temporal  abstraction  was  implemented  in  the  interest  of 
simplicity,  as  well  as  to  more  closely  align  with  standard  Air  Force  availability  planning 
factors  of  20.9  assigned  days  per  month  (AFI  38-201,  2003).  The  figure  of  7.5  months 
was  selected  based  on  a  standard  of  6  months  for  typical  manning  studies  (Juarez,  2010), 
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with  additional  time  added  to  allow  for  data  truncation.  Pre-experiment  tests  revealed  no 
definitive  warm-up  period,  which  was  determined  most  likely  to  be  a  result  of  the 
cyclical  nature  of  the  simulation.  However,  since  time-keeping  logic  within  the  model 
allowed  for  6  days  of  flying  the  first  simulated  week,  this  entire  week  was  truncated  prior 
to  commencement  of  any  analysis.  A  variance  assessment  was  performed  on  the 
responses  of  interest,  and  it  was  determined  that  after  20  replications,  variance  remained 
relatively  constant;  thus,  25  replications  per  experimental  treatment  were  used. 

Initial  results  from  the  model  were  disappointing,  portraying  utilization  rates  of 
less  then  1%  for  many  of  the  specialist  AFSCs,  and  resulting  in  an  almost  universal  lack 
of  statistical  significance  in  the  analysis  of  variance  performed  on  the  experiment  results. 
It  was  determined  that  the  model’s  current  logic  of  assigning  a  single  break  instance  each 
time  that  a  break  was  determined  to  have  occurred  was  the  culprit.  Since  the  specialist 
AFSCs  were  historically  tasked  less  according  to  the  fitted  data,  certain  AFSCs  were  not 
receiving  sufficient  taskings  throughout  any  day  to  induce  any  stress  on  that  specific  job 
type.  A  modification  was  made  to  the  logic,  enabling  a  stochastically  determined  number 
of  break  instances  to  occur  each  time  a  break  was  determined  to  have  happened.  Again, 
two  years  of  data  were  utilized  to  form  two  separate  distributions:  one  for  breaks 
occurring  in  flight  and  another  for  breaks  identified  during  ground  inspections. 

With  these  modifications  in  place.  Table  2  depicts  the  results  of  the  screening 
experiment,  which  were  somewhat  at  odds  with  the  original  expectations.  However, 
upon  further  reflection,  a  plausible  explanation  was  determined. 
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Table  2  -  Screening  Test  Resnlts 


Responses 

Significant 
Facto  r(s) 

Curvature? 

Responses 

Significant 

Factor(s) 

Curvature? 

CCS  UTE 

CC,  AV 
Manning 

No 

EES  UTE 

EE  Manning 

No 

CCS  UTE 

CC  Manning 

Yes 

EES  UTE 

CC,  AV,  EE 
Manning 

No 

CC7  UTE 

CC,  AV 
Manning 

No* 

EE7  UTE 

EE  Manning 

Yes 

AV3  UTE 

AV  Manning 

Yes 

JET  S  UTE 

CC  Manning 

No 

AV5  UTE 

AV  Manning 

Yes 

JET  5  UTE 

JET  Manning 

No* 

AV7  UTE 

AV  Manning 

No 

JET  7  UTE 

N/A 

No 

Sorties/Week 

N/A* 

No 

Cancels/Week 

N/A* 

No 

Eegend:  CC  -  Crew  Chiefs,  AV  -  Avionics,  EE  -  Electro-Environmental,  JET  -  Propulsion 
*  -  P-value  for  significance  (F-test)  greater  than  0.05,  but  less  than  0.1 

First,  examine  the  emergenee  of  multiple  signifieant  faetors  for  various  individual 
responses,  looking  speoifieally  at  3-level  erew  ehief  utilization  as  an  example.  In  this 
ease,  it  is  arguable  that  even  though  eaeh  AFSC  works  independently,  a  signifieant 
number  of  crew  chief  and  avionics  jobs  might  be  needed  in  extended  maintenance  on 
broken  aircraft,  especially  if  either  or  both  of  these  manning  pools  was  affected  in  some 
manner.  This  would  result  in  an  elevated  maintenance  priority  for  aircraft  slated  to  fly, 
which  would  then  prohibit  the  completion  of  any  training,  the  end  result  being  a  decrease 
in  the  3-level  utilization  rate.  Similarly,  the  appearance  of  some  curvature  within  many 
of  the  responses  was  expected,  since  reductions  or  additions  to  the  pool  of  available 
manpower  should  result  in  the  remainder  of  the  available  manpower  pool  working  more 
or  less,  respectively,  in  order  to  keep  up.  These  results  allowed  for  a  reduction  in  scope 
of  the  experiment  to  a  2"^  full  factorial,  reducing  the  overall  testing  to  less  than  a  quarter 
of  the  initial  design  expectation. 
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2,6  Results  and  Analysis 


The  running  of  the  remainder  of  the  2“^  full  factorial  experiment  served  to  further 
solidify  the  assessments  made  on  the  initial  screening  experiment.  While  certain  AFSCs 
and  skill  levels  exhibited  multiple  significant  factors,  each  of  these  is  easily  attributable 
to  causes  similar  to  that  discussed  above.  Results  from  the  2“^  full-factorial  experiment 
are  included  below  in  Table  3.  Changes  from  the  initial  screening  experiment  results  are 
indicated  by  underlined  and  italicized  text  within  the  table. 


Table  3  -  Results  of  Full  2'*  Experimeut 


Responses 

Significant 
Facto  r(s) 

Curvature? 

Responses 

Significant 

Factor(s) 

Curvature? 

CCS  UTE 

CC,AV* 

Manning 

No 

EES  UTE 

EE  Manning 

No 

CCS  UTE 

CC  Manning 

Yes 

EES  UTE 

CC,AV*,  EE, 
JET*  Manning 

No 

CC7  UTE 

CC,AF* 

Manning 

Yes 

EE7  UTE 

CC.  AV,  EE 
Manning 

Yes 

AV3  UTE 

AV  Manning 

Yes 

JET  S  UTE 

AVJET* 

Manning 

No 

AV5  UTE 

AV,  JET* 
Manning 

Yes 

JET  5  UTE 

CC.  JET  Manning 

No 

AV7  UTE 

AV  Manning 

No 

JET  7  UTE 

CC*  Manning 

No 

Sorties/Week 

AV  Manning 

No 

Cancels/Week 

AV  Manning 

No 

Legend:  CC  -  Crew  Chiefs,  AV  -  Avionics,  EE  -  Electro-Environmental,  JET  -  Propulsion 
*  -  P-value  for  significance  (F-test)  greater  than  0.05,  but  less  than  0.1 

Similar  to  the  initial  results  discussed  above,  the  results  indicate  a  surprising 
number  of  relationships  between  individual  AFSCs  that  were  not  predicted  to  have 


36 


existed.  An  initial  coneern  was  that  the  results  might  be  due  to  some  invalidity  of  the 
fundamental  distributional  assumptions  for  the  analysis  of  variance.  Other  than  some 
slight  departures  from  normality  in  the  tails  of  the  analyzed  residuals,  however,  the 
underlying  assumptions  of  normally  and  independently  distributed  errors  with  constant 
variance  were  verified  to  hold.  Also,  as  one  considers  the  effects  of  dynamic 
reprioritization  of  maintenance,  it  becomes  easier  to  visualize  that  these  are  the  apparent 
effects  of  immutable  production  requirements  being  levied  upon  a  dynamic  grouping  of 
resources.  In  a  “real-world”  sense  this  represents  a  unit’s  production  staff  waging  their 
day-to-day  battle  of  meeting  the  flying  mission  while  simultaneously  attempting  to 
provide  sufficient  training  opportunities  to  junior  troops.  As  available  qualified  resources 
become  scarce,  they  are  forced  to  sacrifice  training  in  order  to  maintain  levels  of 
production  necessary  to  meet  mission  requirements.  The  case  of  the  EE5  LITE  response 
provides  an  excellent  illustration  of  these  types  of  effects  and  interactions.  As  evidenced 
in  Eigure  10,  as  individual  AESC  manning  levels  are  modified,  the  resulting  effects  on 
EE  utilization  are  unmistakable.  Additional  details  on  the  analysis  performed  can  be 
found  in  Appendix  A. 
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Figure  10  -  EES  UTE  Response  Values 


Figure  1 1  provides  a  graphical  comparison  of  weekly  cancellations  over  time 
between  the  reduced  manning  (left)  and  increased  manning  (right)  scenarios.  In  the 


Figure  11  -  Comparison  of  Cancels  per  Week 
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reduced  case,  notice  that  the  slope  of  the  data’s  average  is  relatively  flat,  indicating  that 
maintenance  is  in  “survival  mode”,  essentially  striving  to  maintain  one  specific  level  of 
performance.  Conversely,  the  increased  manning  scenario  provides  the  capacity  for 
training  of  junior  personnel  to  occur  with  greater  regularity,  which  results  in  a  net 
increase  in  capacity  as  the  unit’s  overall  average  skill  level  increases.  The  end  result  is  a 
decrease  in  overall  cancellations  per  week  as  time  goes  by. 

2.7  Conclusions 

The  nature  of  the  results  presented  provide  a  level  of  fidelity  heretofore 
unavailable  from  both  current  and  past  methodologies  surveyed  in  terms  of  details  on 
individual  skill  levels.  Since  many  of  the  processes  external  to  the  core  sortie  generation 
process  were  abstracted  out  of  the  model,  it  might  be  considered  fruitless  to  conjecture  on 
the  efficacy  of  the  specific  utilization  rates  produced  by  the  model.  However, 
considering  the  model  presents  a  “best  case”  scenario  in  which  maintenance  personnel 
are  required  to  deal  only  with  the  unscheduled  maintenance  items  that  crop  up  on  a  day  to 
day  basis,  this  model  provides  significant  insight  into  the  relationships  between  specific 
AFSCs  and  skill  levels  and  their  effects  both  on  AFSC  utilization  as  well  as  sortie 
production  capacity. 

In  summary,  it  is  clear  not  only  that  differing  mixes  of  skills  within  individual 
AFSCs  can  exert  significant  influence  on  a  unit’s  capacity  and  capability,  but  also  that 
the  use  of  an  agent  base  modeling  framework  is  effective  in  capturing  many  of  the 
dynamic  relationships  that  drive  the  complex  processes  involved  in  sortie  generation. 
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3,  Case  Study 


Assessing  Maintenance  Capability 


3,1  Introduction 

In  the  summer  of  2008,  after  notieing  a  highly  variant  trend  in  the  eombat  mission 
readiness  (CMR)  of  his  assigned  forces,  the  commander  of  United  States  Air  Forces  in 
Europe  (USAFE)  tasked  his  staff  to  examine  all  facets  of  the  CMR  process  and  determine 
what  the  major  causal  factors  were  (Eipina,  2009).  As  a  result  of  this  tasking,  a 
regression  study  was  completed  in  2009  that  quantified  a  variety  of  factors  driving  the 
variation  in  the  CMR  metric  (Eipina,  2009).  Based  on  the  study’s  results,  which  were 
briefed  to  the  Air  Force  Chief  of  Staff  in  the  spring  of  2009,  additional  taskings  were 
levied  on  the  operations  (AF/A3)  and  logistics  (AF/A4)  directorates  of  the  Air  Staff. 
While  AF/A3  embarked  on  an  enterprise  assessment  of  the  CMR  metric’s  composition, 
documentation  and  reporting  requirements,  AF/A4  was  tasked  to  examine  and  address  the 
use  of  aircraft  maintenance  capability  metrics  within  the  Air  Force.  Specifically,  it  was 
identified  that  no  standard  definition  of  maintenance  capability  existed,  and  the  current 
methods  and  models  used  across  the  Air  Force  fail  to  adequately  and  convincingly 
capture  the  effects  of  maintenance  capability  on  production  capacity  (AF/A4F,  2009). 

This  paper  is  organized  as  follows.  A  brief  literature  review  on  related  work  on 
maintenance  capability  and  production  capacity  is  provided  in  Section  3.2.  An  optimal 
modeling  paradigm  is  selected  and  details  on  its  development  are  provided  in  Section  3.3. 
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The  prototype  model  presented  provides  an  original  contribution  to  the  current  body  of 
research,  and  analytical  results  of  a  simple  representative  scenario  are  provided. 

3.2  Background 

There  is  a  significant  body  of  work  that  addresses  multiple  questions  surrounding 
a  maintenance  unit’s  capacity  and  capabilities.  This  research  revealed  a  string  of  studies 
dating  back  almost  30  years,  all  focused  in  some  part  on  a  typical  maintenance  unit’s 
ability  to  successfully  meet  its  operational  requirements.  A  characteristic  theme  within 
all  of  these  studies  is  a  focus  and  evaluation  of  the  units  sortie  generation  ability.  The 
specific  objectives  vary  greatly,  spanning  everything  from  general  system  observation 
and  characterization  to  attempts  to  optimize  one  or  more  constituent  sub-processes  within 
the  overall  sortie  generation  process.  These  research  efforts  have  employed  many 
methods,  including  discrete  event  simulation  (Faas,  2003;  lakovidis,  2005),  Markov 
decision  analysis  (Dietz,  1991)  and  neural  networks  (Dagg,  1991).  Some  of  these  even 
specifically  addressed  the  issue  of  maintenance  capability  and  its  potential  effects  on 
sortie  production  and  overall  readiness  (Gotz  and  Stanton,  1986;  Garcia  and  Racher, 
1981).  Regardless,  the  methodologies  utilized  follow  a  more  traditional  approach  of 
decomposing  the  system  under  investigation  and  attempting  to  describe  its  behavior  as 
the  “sum  of  its  parts”,  which  has  been  shown  to  be  “inadequate  to  model  and  analyze” 
some  large  and  complex  systems  (Kaegi  et  al,  2009).  In  fact,  research  performed  across 
multiple  disciplines  has  shown  that  these  traditional  methods  of  system  decomposition 
and  subsequent  reconstitution  can  prove  not  only  inadequate  but  also  potentially  produce 
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misleading  results  (Bobashev  et  al,  2007).  Kaegi  et  al  (2009)  further  argue  that  in  these 
situations,  agent  based  modeling  (ABM)  has  a  “high  potential  to  help  realistieally  model 
large  and  eomplex  systems”. 

Sortie  generation  involves  an  exeeptionally  eomplex  set  of  proeesses  with  a 
variety  of  stoehastie  elements  and  external  influenees.  Whether  one  eontemplates  the 
individual  electrieian  or  erew  chief  on  the  flight  line,  the  supply  troop  in  the  Logistics 
Readiness  Squadron,  or  even  individual  aircraft  or  parts  being  moved  up  and  down  the 
supply  chain,  each  plays  a  fundamental  role  in  the  overall  process.  Additionally,  while 
the  behaviors  and  motivations  of  each  of  these  constituent  pieces  is  relatively  well 
understood,  the  resulting  behavior  of  the  system  as  a  whole  is  more  complex  than  any 
explanation  any  individual  component  could  provide.  This  is  the  hallmark  of  a  complex 
system  (Flake,  2002),  an  environment  directly  suited  to  an  ABM’s  ability  to  reveal 
“properties  of  systems  that  are  not  properties  of  the  agents  themselves”  (Jones,  2007). 

3,3  An  Agent  Based  Sortie  Generation  Simulation 

Due  to  the  identified  complexity  of  sortie  generation,  a  model  of  this  process 
would  benefit  from  use  of  an  ABM  structure.  Looking  specifically  at  the  inter¬ 
relationships  between  maintenance  personnel  across  a  variety  of  skill  levels  and  job 
specialties  and  their  potential  outputs  in  terms  of  sortie  production,  an  ABM  provides  a 
detailed  individual-based  perspective  on  the  overarching  process.  The  specific  focus  of 
this  research  involves  the  on-equipment  maintenance  portion  of  the  sortie  generation 
process.  The  simulation  model  is  used  to  examine  the  effects  of  various  levels  of 
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maintenance  manning  on  sortie  production  and  manning  utilization  while  taking  into 
account  the  specific  abilities  of  individual  maintenance  personnel  across  a  variety  of  job 
specialties  and  skill  levels. 

3,3.1  Simulation  Model  Development 

Development  of  the  model  required  several  key  assumptions.  These  were:  a) 
flying  window  remains  constant;  b)  sortie  load  retains  a  set  weekly  pattern;  c)  aircraft 
configuration  is  not  a  concern;  d)  scheduled  maintenance  is  not  modeled;  and  e)  collected 
data  used  to  determine  underlying  distributions  is  assumed  to  be  accurate  and 
representative  of  the  underlying  real-world  systems. 

Figure  12  displays  a  top-level  view  of  the  modeled  sortie  generation  process. 
Within  the  model,  four  separate  agent  types  interact  according  to  specific  defined 
behaviors  in  order  to  accomplish  the  tasks  making  up  various  portions  of  the  sortie 
generation  process.  In  brief,  the  defined  agent  types  are:  Production  Supervisor, 
Expediters,  Aircraft  and  Maintenance  agents.  The  Production  supervisor  provides 
general  oversight  and  direction  to  the  other  agents.  The  expediter  agents  allocate 
personnel  to  their  assigned  tasks  and  are  broken  down  into  crew  chief,  avionics  and 
mechanical  (electro-environmental  and  propulsion)  specialties. 
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Figure  12  -  Modeled  Sortie  Generation  Process 


Finally,  the  maintenance  agents  serve  as  assignable  resources  and  are  further 
defined  by  their  AFSC  (crew  chief,  avionics,  electro-environmental  or  propulsion)  and 
skill  level  (3-,  5-  or  7-level).  Each  of  the  maintenance  agents  possesses  a  learning  curve 
which  increases  their  efficiency  (modeled  as  working  speed)  over  time. 


3,3,2  Model  Execution 


The  model  begins  in  an  empty  and  idle  state.  Following  the  general  outline  of  the 
process  depicted  in  Figure  12,  a  sortie  requirement  is  placed  on  the  system,  which  results 
in  the  assignment  of  specific  aircraft  to  sorties  and  an  allocation  of  additional  aircraft 
designated  as  spares.  Jobs  are  assigned  for  each  of  these  aircraft,  which  are  in  turn 
assigned  to  individual  maintenance  crews  by  the  appropriate  expediter.  Depending  on 
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the  priority  assigned  to  eaeh  job  by  the  pro  super,  some  jobs  are  opened  as  training 
opportunities  for  junior  (3-level)  maintenanee  troops.  As  jobs  are  completed,  the  released 
aircraft  will  fly  their  assigned  sorties,  return  and  be  prepped  for  follow-on  missions  or 
receive  their  end-of-day  inspections.  This  cycle  is  continuously  repeated,  with  sorties 
being  scheduled  5  days  a  week,  and  maintenance  crews  operating  on  3  rotating  8-hour 
shifts. 

Breaks  are  stochastically  determined  at  various  points  within  the  model.  Each 
ground  inspection  carries  a  chance  to  uncover  an  issue,  and  each  sortie  has  the  potential 
for  either  an  abort  (immediate  sortie  failure)  or  Code  3  break  (sortie  completion,  but 
aircraft  lands  with  maintenance  issue).  At  each  of  these  junctures,  a  random  draw 
determines  the  number  of  issues  found,  and  each  of  their  associated  WUCs.  All 
maintenance  crew  sizes  and  fixtime  lengths  for  each  task  are  stochastically  determined 
via  a  multi-branch  conditional  tree.  Conditioned  on  the  identified  WUC,  separate  trees 
identify  both  an  AFSC  and  then  a  fitted  theoretical  distribution  from  which  a  fix  time  is 
drawn.  Finally,  an  assessment  is  made  of  the  assigned  team’s  mean  efficiency  rating, 
which  is  then  used  to  modify  the  drawn  fix  time.  The  net  effect  is  lengthened  fix  times 
for  lower  efficiencies,  with  the  unit’s  mean  efficiency  (and  thus  speed  of  work) 
increasing  over  time  as  individual  maintenance  agents  learn  their  jobs.  All  distributions 
were  based  on  2  years  of  data  from  Spangdahlem  AB,  Germany. 
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3,4  Analysis 


This  section  provides  an  analysis  of  the  model’s  output  when  populated  with  a 
typical  maintenance  unit’s  maintenance  manning  levels.  An  additional  response, 
maintenance  efficiency,  is  also  discussed. 

3,4,1  Experimental  Design 

Fourteen  core  responses  were  identified  as  outputs  of  interest  from  the  model. 
Twelve  responses  were  the  daily  utilization  rates  of  each  AFSC  and  skill  level,  and  the 
final  two  were  the  weekly  figures  for  sorties  produced  and  cancellations.  An  additional 
set  of  responses,  maintenance  efficiencies,  were  also  analyzed.  Each  agent  has  a 
maintenance  efficiency  varied  from  0  (no  skill)  to  1  (highly  skilled).  This  measured  the 
average  increase  in  efficiency  across  each  AFSC  and  skill  level,  capturing  the  effects  of 
varied  manning  availability  scenarios  on  the  ability  of  a  unit  to  continue  to  train  and 
develop  its  junior  maintenance  troops.  Four  factors  of  interest  were  identified  to  drive 
the  experiment:  manning  levels  for  each  of  the  four  AFSCs  modeled.  A  full-factorial  2“^ 
experiment  was  executed.  Using  a  baseline  level  of  manning  modeled  after  data  gathered 
from  Shaw  AFB  as  centerpoints,  high  and  low  test  levels  were  formulated  based  on  a 
10%  increase  or  reduction  in  available  manning  for  each  AFSC. 
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3,4,2  Results 


Test  results  were  based  on  a  run  length  providing  slightly  over  7  months  of  data 
after  deleting  the  first  week.  While  the  majority  of  AFSCs’  learning  curves  were  based 
on  a  12  month  developmental  cycle,  this  shorter  time  span  allowed  for  additional 
replications  and  test  points  while  still  providing  a  solid  indication  of  increased 
proficiency.  After  some  initial  testing,  it  was  determined  that  output  variance  stabilized 
after  20  replications;  25  replications  per  design  point  was  used.  Post-test  evaluation  of 
the  responses  indicated  no  severe  departures  from  normality,  so  a  series  of  standard 
ANOVA  tests  was  performed  for  each  of  the  responses.  Results  are  provided  in  Table  4. 


Table  4  -  Experiment  Results 


Responses 

Significant 

Factor(s) 

Curvature? 

Responses 

Significant 

Factor(s) 

Curvature? 

CCS  UTE 

CC,  AV* 
Manning 

No 

EES  UTE 

EE  Manning 

No 

CCS  UTE 

CC  Manning 

Yes 

EES  UTE 

CC,  AV*,  EE, 
JET*  Manning 

No 

CC7  UTE 

CC,  AV* 
Manning 

Yes 

EE7  UTE 

CC,  AV,  EE 
Manning 

Yes 

AV3  UTE 

AV  Manning 

Yes 

JET  S  UTE 

AV,  JET* 
Manning 

No 

AV5  UTE 

AV,  JET* 
Manning 

Yes 

JET  5  UTE 

CC,  JET 
Manning 

No 

AV7  UTE 

AV  Manning 

No 

JET  7  UTE 

CC*  Manning 

No 

Sorties/Week 

AV  Manning 

No 

CancelsAVeek 

AV  Manning 

No 

Eegend:  CC  -  Crew  Chiefs,  AV  -  Avionics,  EE  -  Electro-Environmental,  JET  -  Propulsion 
*  -  P-value  for  significance  (F-test)  greater  than  0.05,  but  less  than  0.1 
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While  the  results  presented  in  Table  4  are  based  on  the  varied  numerieal  results 
provided  from  the  model,  the  numbers  are  not  the  response  of  interest.  With  the  variety 
of  abstraetions  utilized  to  develop  the  model,  the  utility  of  these  numerieal  responses  as 
point  estimators  for  true  system  performanee  is  questionable.  Instead,  the  surprising 
number  of  dependeneies  identified  between  disparate  groups  of  manning  is  of  key 
interest,  ehiefly  beeause  no  interaction  between  any  of  these  groups  was  included  as  a 
part  of  the  model  logic. 

Using  the  EE5  UTE  response  as  an  example,  the  original  assumption  was  that  the 
only  factor  of  significance  would  be  the  EE  manning  pool.  However,  when  one  considers 
that  the  CC  and  AV  AESCs  received  the  highest  number  of  taskings,  or  that  the  average 
JET  fix  times  tend  to  be  somewhat  lengthy,  it  is  easy  to  see  that  modifications  to  the 
available  manning  for  any  of  these  AESCs  might  drive  maintenance  timelines.  Coupled 
with  the  fact  that  job  priorities  upgrade  automatically  if  timelines  were  not  being  met,  and 
potentially  prohibit  completion  of  any  training,  one  can  see  that  this  could  lead  to 
shortened  job  times  since  the  trainers  (5-  and  7-levels)  are  no  longer  slowing  down  work 
efforts  in  order  to  train  lower  skill-level  members. 

A  separate  evaluation  was  performed  on  the  effects  on  efficiency  gains  as  a  result 
of  differing  manning  levels.  Eigure  13  depicts  results  from  the  extreme  cases,  comparing 
the  baseline  case  to  the  reduction  and  addition  of  10%  for  all  AESCs  at  once.  The  chart 


48 


9.0% 

I  8.0% 

OJ 

£;  7.0% 

6.0% 
c  5.0% 
4.0% 
£  3.0% 

M  2.0% 

5  1.0% 
<  0.0% 


Figure  13  -  Comparison  of  Average  3-level  Efficiency  Increase 


serves  as  a  reminder  of  the  additional  complexities  within  the  modeled  system.  In  this 
case,  the  seemingly  counterintuitive  reduction  in  overall  efficiency  gains  is  actually  a 
result  of  the  additional  manning  added  to  shifts  where  work  load  was  not  as  heavy.  As 
such,  these  results  are  largely  colored  by  circumstances  peculiar  to  the  simulation,  but 
also  underscore  the  importance  of  proper  shift  manning,  in  addition  to  the  overarching 
import  of  sufficient  manning  levels  unit-wide. 


3,5  Conclusions 


This  model  and  analysis  has  examined  the  effects  of  varied  levels  of  manning 
availability  on  both  the  utilization  and  production  capacity  of  a  maintenance  unit.  As 
previously  mentioned  it  is  not  the  intent  to  assign  specific  significance  to  the  numerical 
results  presented,  but  instead  to  the  insight  gained  on  the  multiple  layers  of  interactions 
between  the  various  AFSCs  and  skill  levels  in  executing  the  sortie  generation  process. 
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The  results  presented  indicate  a  series  of  complex  interactions  between  the 
various  groups  that  make  up  the  primary  execution  arm  of  the  sortie  generation  process. 
Analysis  shows  that  even  slight  changes  in  the  structure  of  available  manning  can  have 
significant  effects  on  a  unit’s  maintenance  capacity  and  ability  to  develop  over  time. 

This  has  direct  implications  to  any  maintenance  unit’s  capability  to  sustain  and  support  a 
wing’s  CMR  requirement. 
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4.  Conclusion 


4.1  Research  Summary 

This  thesis  developed  a  novel  simulation  methodology  eapturing  the  sortie 
generation  proeess.  It  eaptured  the  eore  sortie  generation  proeesses  and  methods,  and 
integrated  these  within  a  eonstruet  of  behavior-driven  adaptable  entities.  This  agent 
based  methodology  provides  details  on  a  variety  of  system  player  interaetions  not 
available  through  methods  eurrently  being  employed. 

A  representative  seenario  based  on  eurrent  manning  levels  and  historie 
maintenanee  data  was  performed.  While  the  seope  of  this  prototype  model  preeludes 
direet  validation  of  the  numerie  output,  aeouraey  of  the  internal  model  logie  and  behavior 
models  were  validated  via  multiple  interviews  with  eareer  maintenanee  personnel  at 
Shaw  AFB,  Wright  Patterson  AFB  and  Spangdahlem  AB. 

4.2  Future  Work 

An  effort  is  eurrently  underway  within  the  A4  direetorate  of  the  Air  Staff  to 
examine  the  methodologies  used  to  model  unit  level  maintenanee  eapaeity  and  eapability, 
and  evaluate  whether  a  new  method  might  be  warranted.  As  mentioned  in  Chapter  2, 
despite  its  high  level  of  detail,  the  LCOM  has  multiple  identified  shorteomings  that 
detraet  from  its  use  for  this  speeifie  applieation.  This  thesis  has  demonstrated  the 
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capability  of  an  agent-based  model  to  eapture  some  of  the  eritieal  interaetions  within  a 
unit  and  their  effeets  on  that  unit’s  eapacity  both  to  produee  and  sustain  produetion  over 
time.  As  a  unit’s  net  effieieney  ehanges  over  time  due  to  retention  issues,  deployments 
and  permanent  ehanges  of  station,  an  understanding  of  the  related  potential  effects  on  a 
unit’s  capability  becomes  eritieal.  This  is  espeeially  true  in  an  era  when  our  end  strength 
is  eonstrained  and  every  unit  is  expeeted  to  do  more  with  less:  a  skill-balanced  force  is 
vital. 

The  model  as  presented  would  definitely  benefit  from  some  expansion.  Three 
speeifie  areas  identified  during  the  eourse  of  model  development  and  analysis  were:  task 
detail,  seope  and  the  fidelity  of  agent  behavior  models.  As  mentioned  in  Chapter  2,  the 
questionable  validity  of  available  maintenanee  data  makes  it  difficult  to  assess  the  overall 
validity  of  the  numerieal  outputs  of  the  model.  The  LCOM  has  overeome  this  in  some 
regards  by  eapturing  detailed  network  flows  eapturing  task  level  detail  for  speeifie  jobs 
down  to  the  5-digit  WUC.  Integrating  this  level  of  detail  into  the  model  would 
eonsiderably  improve  the  overall  validity  of  the  produet.  Seeond,  having  insight  into  the 
variety  of  seheduled  inspections  performed  and  the  effects  on  personnel  and  aireraft 
availability  would  add  a  valuable  layer  of  realism  to  the  model.  Finally,  while  even  the 
simple  behavior  models  in  plaee  were  shown  to  be  effective,  behaviors  sueh  as  eross- 
utilization  training  (maintainors  able  to  work  on  tasks  outside  their  eore  AFSC  in  order  to 
more  effeetively  share  work  load)  would  be  useful  to  inelude,  espeeially  as  the  other  two 
items  are  addressed,  and  work  load  requirements  inerease  aeeordingly  within  the  model. 

Finally,  while  the  model  developed  for  this  thesis  is  easily  adaptable  to  virtually 
any  weapon  system  within  the  Air  Foree  inventory,  it  remains  a  model  of  the  sortie 
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generation  proeess.  However,  the  same  eoneepts  employed  ean,  and  have  been, 
employed  in  a  variety  of  other  environments  comprised  of  complex  sets  of  relationships 
or  behaviors.  As  a  sociology  tool,  it  is  well-suited  to  the  analysis  of  human  systems. 

This  suggests  utility  in  a  variety  of  manning  studies,  and  is  evidenced  by  a  variety  of 
work  in  this  specific  area.  Recent  work  of  note  include  Hill  and  Gaupp’s  use  of  agents  in 
modeling  the  pilot  retention  problem  for  the  USAF  (2006),  or  a  similar  study  using 
agents  to  model  the  Navy’s  manpower  system  (Trifonov  et  al,  2005). 


53 


Appendix  A.  Analysis  Details 


This  section  contains  additional  details  on  specific  methodology  and  results 
employed  throughout  this  thesis.  It  is  separated  into  developmental  and  analytical  details. 

Al.  Development  Methodology 

Any  modeling  of  the  sortie  generation  process  becomes  a  data  intensive  endeavor. 
Given  the  documented  difficulties  in  obtaining  “clean”  data,  great  pains  were  taken  to 
attempt  to  ensure  that  the  data  utilized  was  as  accurate  and  representative  of  truth  as 
possible.  With  the  plethora  of  data  available,  this  involved  both  the  selection  of 
appropriate  data,  as  well  as  a  series  of  filtering  processes. 

The  first  step,  data  selection,  involved  choosing  the  types  of  data  to  be  included 
for  formulating  the  variety  of  distributions  required  within  the  model.  Data  was 
partitioned  based  on  whether  it  was  a  result  of  a  ground-determined  or  flight-found  issue. 
This  partition  was  enabled  through  the  use  of  when-discovered  (WD)  codes  contained 
within  the  data  records.  Codes  and  associated  definition  for  ground-based  data  are  as 
follows  (TO  00-20-7,  2007): 

E  -  After  flight 

F  -  Between  flight  -  ground  crew 
H  -  Thru-flight  inspection 
J  -  Preflight  Inspection 
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Similarly,  for  flight-related  data,  codes  and  definitions  utilized  were  (TO  00-20-7,  2007): 

C  -  In-flight,  Abort 

D  -  In-flight,  No  Abort 

The  second  and  final  step  involved  a  series  of  filter  operations.  After  gathering  the 
two  macro  sets  of  data,  it  was  discovered  that  despite  having  only  collected  data  coded  as 
unscheduled  maintenance  actions,  a  great  deal  of  individual  records  dealt  with  regularly 
scheduled  inspection  items.  Luckily,  the  vast  majority  of  these  were  coded  under  a  single 
how-malfunctioned  code,  used  to  indicate  “how  or  why  a  piece  of  equipment 
malfunctioned”  (TO  00-20-2,  2007).  Data  was  further  reduced  to  remove  any  record 
associated  with  this  code.  Additionally,  there  were  many  cases  where  multiple  records 
existed  for  the  accomplishment  of  a  single  maintenance  issue.  While  in  reality  this  would 
result  from  completion  of  a  variety  of  tasks  associated  with  a  single  break,  the  presence 
of  multiple  records  served  to  positively  bias  some  AFSCs  assignment  distributions,  while 
negatively  biasing  their  fix  time  distribution.  Therefore,  an  automated  filter  was  run 
through  the  data  set  that  condensed  multiple  identical  line  items  (based  on  individual  job 
control  number)  into  one  record,  and  assigned  its  fix  time  as  the  sum  of  all  constituent 
records.  This  served  to  more  effectively  capture  the  time  worked  per  issue  and  avoid  the 
biases  previously  presented  by  the  data  structure. 
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A2,  Analysis  Results 


Additional  numerical  results  and  specific  details  on  the  analytic  process  are 
included  here  for  completeness.  Output  analysis  methods  and  ANOVA  results  are 
discussed.  In  the  interest  of  brevity,  only  a  few  representative  examples  are  presented. 

As  previously  mentioned,  each  simulation  run  provides  over  7  months  of 
analyzable  data.  A  month  of  simulation  time  was  considered  to  be  4  weeks,  or  28  days. 
While  no  transient  was  discerned  after  evaluating  50  replications  of  the  model,  a  logic 
issue  allows  for  flights  to  occur  6  days  over  the  first  week.  Thus,  data  from  the  first  week 
was  truncated,  leaving  29  weeks  of  usable  data  for  subsequent  analysis. 

Data  for  each  of  the  responses  was  first  organized  to  capture  the  m  replications  of 
n  realizations  for  each  of  the  output  sequences  {X^:  n  =  1,2,  3, ... }.  This  yields  a  matrix 
of  the  form 


All 

■ 

Aijv 

^21 

^22 

^2N 

^M1 

^M2 

In  the  case  of  the  12  AFSC  and  skill  level  UTE  responses,  since  the  daily  UTE 
rate  was  more  of  a  concern  than  the  weekly  rate,  Welch’s  method  of  replications  (Welch, 
1983)  was  applied,  yielding  a  grand  mean  for  the  entire  data  set  representing  the  average 
daily  utilization  rate  for  each  AESC  and  skill  level.  A  similar  method  was  employed  for 
the  collected  flight  data,  but  calculated  so  as  to  provide  an  average  weekly  figure  for 
each. 
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ANOVA  tests  were  performed  for  each  of  the  identified  responses.  An  excellent 


illustrative  example  is  the  5-level  EE  LITE  response  (EE5  LITE).  Since  the  primary 
interest  was  in  identifying  significant  interactions  between  agents  within  the  model,  no 
specific  level  of  a  was  selected  for  evaluation  of  the  model.  Instead,  those  items 
determined  significant  at  the  a=.05  level  were  immediately  marked  as  significant,  while 
those  significant  at  the  a=0.1  level  were  considered  within  a  range  of  significance  of 
interest  to  this  analysis.  The  p-values  for  factors  qualifying  under  the  former  criteria  are 
highlighted  in  Table  5  below,  while  those  qualifying  at  the  latter  criteria  are  marked  with 
an  asterisk. 


Table  5  -  EES  UTE  ANOVA  Results 


Source 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Value 

p-value  Prob  > 

F 

Model 

0.013 

4 

3.18E-03 

36.48 

<  0.0001 

A-CC  Manning 

5.77E-04 

1 

5.77E-04 

6.62 

0.0105 

B-AV  Manning 

2.88E-04 

1 

2.88E-04 

3.3 

0.0701* 

C-EE  Manning 

0.012 

1 

0.012 

132.69 

<  0.0001 

D-JT  Manning 

2.91E-04 

1 

2.91E-04 

3.34 

0.0684* 

Curvature 

9.87E-05 

1 

9.87E-05 

1.13 

0.2883 

Residual 

0.037 

419 

8.73E-05 

Lack  of  Fit 

8.18E-04 

11 

7.44E-05 

0.85 

0.5915 

Pure  Error 

0.036 

408 

8.76E-05 

Cor  Total 

0.049 

424 

Critical  ANOVA  assumptions  of  error  normality,  independence  and  constant 
variance  were  investigated  as  a  portion  of  the  analytical  process.  Various  diagnostic 
residual  plots  are  included  in  Eigures  14  through  16  in  the  interest  of  thoroughness. 
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Figure  14  -  EES  UTE  Normal  Probability  Plot 
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Figure  15  -  EES  UTE  Residuals  vs.  Run 


Figure  16  -  EES  UTE  Residuals  vs.  Predicted 
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The  above  series  of  figures  indieate  all  of  the  model  assumptions  hold.  Of 
interest  is  the  apparent  outlier  present  on  eaeh  of  the  figures.  After  debating  whether  or 
not  to  remove  this  datapoint  from  the  overall  dataset,  it  was  decided  to  leave  it  in  place. 
Since  it  was  one  of  the  initial  runs,  presenting  the  worst  case  scenario  with  all  AFSCs 
reduced  by  10%,  it  was  felt  that  this  portrayed  the  very  real  potential  for  a  “snowball 
effect”.  This  occurs  when  a  series  of  bad  breaks  within  a  unit’s  fleet  of  aircraft  end  up 
overwhelming  a  unit’s  ability  to  maintain  their  current  production  rate.  Since  the 
potential  for  this  to  occur  would  be  increased  by  a  loss  in  manning,  it  was  felt  that  this 
point,  although  a  statistical  outlier,  was  indicative  of  potential  system  performance. 

The  results  indicate  a  surprising  level  of  interconnectedness  between  each  of  the 
AFSCs.  While  the  operational  implications  and  potential  causes  were  discussed,  it  is 
useful  to  examine  these  results  from  a  more  objective  standpoint.  It  is  relatively  trivial  to 
understand  the  high  level  of  significance  attributed  to  EE  manning. 

Conversely,  the  factor  with  the  next  highest  level  of  significance,  crew  chiefs, 
might  initially  be  surprising  to  see.  However,  one  must  consider  that  crew  chiefs  are  the 
highest  tasked  AESCs  within  the  model,  receiving  an  expected  43%  of  ground-found 
tasks  and  58%  of  flight-determined  tasks.  With  high  levels  of  taskings,  changes  in 
available  crew  chief  manning  has  the  ability  to  affect  the  priority  of  maintenance  within 
the  model,  which  could  then  affect  whether  training  is  a  possibility.  This  could  directly 
affect  the  utilization  rates  of  5 -level  EE  troops,  who  as  qualified  maintainers  serve  as 
trainers  within  their  AESC,  and  would  therefore  work  slower  or  faster  (on  average) 
depending  on  whether  they  were  involved  in  training  or  not.  Similar  arguments  can  be 
postulated  for  the  other  factors  of  significance. 
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To  determine  whieh  experimental  treatments  exhibited  signifieanee  for  eaeh 
response,  a  comparison  of  each  of  the  experiment  treatments  was  made  to  the  baseline 
(centerpoints).  Using  Dunnett’s  test  as  outlined  in  Montgomery  (2009),  the  test  evaluates 
the  hypothesis 

=  y-a 

Hi- yi  ^  ya 

for  some  /  =  1 ,  2,  . . . ,  a- 1 .  For  this  case,  the  centerpoint  treatment  mean  was  treated  as  the 
control  to  which  the  remaining  treatment  means  were  compared. 

Table  6  provides  a  partial  set  of  results  from  the  test.  Each  of  the  figures 
represents  the  baseline  treatment  mean  subtracted  from  a  specific  treatment  mean. 
Significant  items  (at  the  a=.05  level)  are  depicted  in  bold.  With  the  crew  chief  responses, 
almost  every  single  treatment  was  indicated  as  significantly  different  from  the  baseline 
treatment.  This  would  imply  that  this  AFSC  is  volatile  in  terms  of  its  utilization  rate, 
responsive  to  even  very  small  changes  in  manning  allocations.  As  you  move  to  the  right 
within  the  table,  you  can  see  that  there  is  less  significance  indicated  for  the  avionics 
AFSC.  We  postulate  that  this  is  due  primarily  to  the  difference  in  the  level  of  taskings 
between  these  two  AFSCs,  with  avionics  receiving  5-25%  fewer  taskings  than  crew 
chiefs  depending  on  the  source  of  the  break.  This  is  supported  by  the  fact  that  the  other 
AFSCs  responses,  which  jointly  account  for  16%  and  9%  of  ground  and  flight  taskings, 
respectively,  exhibit  almost  no  significance  due  to  this  test. 


61 


Table  6  -  Results  of  Dunnett's  Test 


Treatment 

CC3 

CC5 

CC7 

AV3 

AV5 

AV7 

1 

0.003838 

0.045307 

0.037123 

0.012954 

0.005724 

0.012893 

2 

0.007201 

0.039957 

0.039339 

0.0144 

0.006188 

0.014324 

3 

0.007885 

0.044204 

0.039502 

0.011525 

0.009378 

0.016102 

4 

0.007566 

0.045523 

0.037383 

0.014716 

0.004258 

0.013049 

5 

0.005025 

0.052863 

0.036039 

0.001286 

0.021979 

0.015769 

6 

0.007066 

0.043567 

0.045657 

0.002579 

0.022144 

0.014941 

7 

0.012893 

0.046145 

0.041664 

0.000232 

0.019682 

0.013918 

8 

0.009032 

0.042881 

0.045638 

0.00237 

0.021737 

0.014137 

10 

0.008953 

0.037366 

0.025625 

0.014952 

0.007881 

0.015579 

11 

0.008469 

0.034808 

0.029719 

0.008811 

0.001719 

0.011289 

12 

0.011369 

0.032867 

0.032721 

0.012128 

0.004517 

0.01289 

13 

0.009914 

0.033655 

0.029855 

0.01051 

0.003939 

0.013521 

14 

0.005448 

0.035527 

0.027269 

0.001314 

0.020217 

0.015094 

15 

0.008625 

0.035221 

0.027366 

0.002627 

0.021216 

0.01489 

16 

0.009233 

0.032295 

0.030219 

0.000935 

0.022479 

0.016641 

17 

0.008689 

0.033102 

0.029364 

0.000951 

0.021372 

0.012945 

Specific  point  estimates  for  each  of  the  responses  returned  from  the  model  were 
not  of  particular  interest  to  the  analysis.  Utilization  rates  ranged  from  roughly  30%  in  the 
case  of  crew  chiefs,  steadily  dwindling  to  under  2%  for  jet  troops.  This  was  not 
surprising  due  to  the  scoping  of  the  model  effort,  which  was  focused  more  on 
determining  the  existence  and  effects  of  the  interactions  between  the  various  agents. 
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Appendix  B,  Selected  Analysis  Results 


This  section  contains  a  selection  of  results  from  the  ANOVA  performed  on  each 
of  the  14  responses  of  interest.  While  not  specifically  included,  note  that  all  necessary 
assumptions  of  residual  normality,  independence  and  constant  variance  were  verified  to 
have  been  met.  Full  results  are  available  upon  request  from  Dr.  Miller  in  AFIT/ENS. 


Table  7  -  CC3  UTE  ANOVA  Results 


Source 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Value 

p-value 
Prob  >  F 

Model 

0.027282 

4 

0.00682 

80.07862 

<0.0001 

A-CC  Manning 

0.026898 

1 

0.026898 

315.8082 

<0.0001 

B-AV  Manning 

0.000317 

1 

0.000317 

3.717429 

0.0545 

C-EE  Manning 

6.67E-05 

1 

6.67E-05 

0.783672 

0.3765 

D-JT  Manning 

4.41E-07 

1 

4.41E-07 

0.005177 

0.9427 

Curvature 

9.55E-06 

1 

9.55E-06 

0.112144 

0.7379 

Residual 

0.035687 

419 

8.52E-05 

Lack  of  Fit 

0.001386 

11 

0.000126 

1.498928 

0.1289 

Pure  Error 

0.034301 

408 

8.41E-05 

Cor  Total 

0.062979 

424 

CC3 


Figure  17  -  CCS  UTE  Cube  Plot 
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Table  8  -  AV7  UTE  ANOVA  Results 


Source 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Value 

p-value 
Prob  >  F 

Model 

0.081267 

4 

0.020317 

282.4728 

<0.0001 

A-CC  Manning 

2.37E-05 

1 

2.37E-05 

0.329299 

0.5664 

B-AV  Manning 

0.08121 

1 

0.08121 

1129.103 

<0.0001 

C-EE  Manning 

3.21E-05 

1 

3.21E-05 

0.44582 

0.5047 

D-JT  Manning 

9.35E-07 

1 

9.35E-07 

0.013003 

0.9093 

Curvature 

6.94E-06 

1 

6.94E-06 

0.096446 

0.7563 

Residual 

0.030136 

419 

7.19E-05 

Eack  of  Fit 

0.000603 

11 

5.48E-05 

0.756745 

0.6834 

Pure  Error 

0.029534 

408 

7.24E-05 

Cor  Total 

0.11141 

424 

AV7 


Figure  18  -  AV7  UTE  Cube  Plot 
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Table  9  -  EE3  UTE  ANOVA  Results 


Source 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Value 

p-value 
Prob  >  F 

Model 

0.004876 

4 

0.001219 

38.13561 

<0.0001 

A-CC  Manning 

7.04E-05 

1 

7.04E-05 

2.200973 

0.1387 

B-AV  Manning 

3.22E-05 

1 

3.22E-05 

1.007541 

0.3161 

C-EE  Manning 

0.004765 

1 

0.004765 

149.0663 

<0.0001 

D-JT  Manning 

8.56E-06 

1 

8.56E-06 

0.267653 

0.6052 

Curvature 

7.17E-06 

1 

7.17E-06 

0.224383 

0.6360 

Residual 

0.013393 

419 

3.2E-05 

Eack  of  Fit 

0.000564 

11 

5.13E-05 

1.631055 

0.0875 

Pure  Error 

0.012829 

408 

3.14E-05 

Cor  Total 

0.018276 

424 

EE3 


Figure  19  -  EE3  UTE  Cube  Plot 
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Table  10  -  JETS  UTE  ANOVA  Results 


Source 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Value 

p-value 
Prob  >  F 

Model 

0.001583 

4 

0.000396 

11.20796 

<0.0001 

A-CC  Manning 

0.00014 

1 

0.00014 

3.975037 

0.0468 

B-AV  Manning 

8.68E-06 

1 

8.68E-06 

0.245764 

0.6203 

C-EE  Manning 

4.94E-05 

1 

4.94E-05 

1.400068 

0.2374 

D-JT  Manning 

0.001385 

1 

0.001385 

39.21098 

<0.0001 

Curvature 

9.4E-05 

1 

9.4E-05 

2.663038 

0.1035 

Residual 

0.014795 

419 

3.53E-05 

Eack  of  Fit 

0.000112 

11 

1.02E-05 

0.282918 

0.9888 

Pure  Error 

0.014683 

408 

3.6E-05 

Cor  Total 

0.016473 

424 

JT5 


Figure  20  -  JET  5  UTE  Cube  Plot 
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Table  11  -  Sorties  AN OVA  Results 


Source 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Value 

p-value 
Prob  >  F 

Model 

9.173227 

4 

2.293307 

2.022795 

0.0904 

A-CC  Manning 

0.982223 

1 

0.982223 

0.866363 

0.3525 

B-AV  Manning 

6.011253 

1 

6.011253 

5.302184 

0.0218 

C-EE  Manning 

0.665973 

1 

0.665973 

0.587417 

0.4439 

D-JT  Manning 

1.513779 

1 

1.513779 

1.335218 

0.2485 

Curvature 

0.871103 

1 

0.871103 

0.76835 

0.3812 

Residual 

475.0335 

419 

1.133731 

Eaek  of  Fit 

7.087663 

11 

0.644333 

0.561791 

0.8596 

Pure  Error 

467.9458 

408 

1.146926 

Cor  Total 

485.0778 

424 

Sorties 


Figure  21  -  Sorties  Cube  Plot 
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Table  12  -  Cancels  ANOVA  Results 


Source 

Sum  of 
Squares 

df 

Mean 

Square 

F 

Value 

p-value 
Prob  >  F 

Model 

10.24565 

4 

2.561413 

2.86 

0.0233 

A-CC  Manning 

1.186543 

1 

1.186543 

1.32486 

0.2504 

B-AV  Manning 

6.685918 

1 

6.685918 

7.465306 

0.0066 

C-EE  Manning 

0.692462 

1 

0.692462 

0.773183 

0.3797 

D-JT  Manning 

1.680727 

1 

1.680727 

1.876652 

0.1714 

Curvature 

0.809965 

1 

0.809965 

0.904384 

0.3422 

Residual 

375.2559 

419 

0.895599 

Eaek  of  Fit 

6.54852 

11 

0.59532 

0.658763 

0.7775 

Pure  Error 

368.7073 

408 

0.903694 

Cor  Total 

386.3115 

424 

Canx 


Figure  22  -  Cancels  Cube  Plot 
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Appendix  C,  Blue  Dart 


Improving  Assessments  of  Maintenance  Capacity  and  Capability 

The  ultimate  source  of  combat  capability  resides  in  the  men  and  women  of  the  Air  Force 

-  (AFDD  1,2003) 

“Do  more  with  less”  sums  up  the  current  operating  environment.  With  fiscal 
constraints  driving  hard  decisions,  balancing  various  requirements  in  manpower,  materiel 
and  weapon  systems.  Air  Force  leaders  are  forced  to  rely  on  a  variety  of  projections  and 
analyses  detailing  the  trade-offs  inherent  in  each  potential  choice.  Each  decision  carries  a 
spectrum  of  ramifications,  with  impacts  spanning  the  gamut  from  fiscal  to  readiness. 

In  recent  years,  the  Air  Force  experienced  a  series  of  manning  cuts  as  a  result  of 
Program  Budget  Decision  720  (PBD  720).  These  were  later  shown  to  have  been 
ineffective  in  achieving  their  goal  of  producing  savings  to  finance  a  recapitalization  of 
the  Air  Force  fleet  (Holmes,  2009).  The  resulting  net  loss  in  experience  within  the 
enlisted  aircraft  maintenance  fields  was  irreparable  in  any  immediate  sense,  addressable 
only  through  an  increase  in  enlistments  and  a  flooding  of  the  training  pipelines  with  new 
maintainers.  This  supported  goals  for  an  increased  end-strength,  but  was  unable  to 
address  the  issue  of  lost  maintainer  experience. 

What  effect  does  such  a  shift  in  a  maintenance  unit’s  net  experience  level  have  on 
its  capacity  to  safely  and  effectively  produce  sorties  and  maintain  sufficient  fleet  health 
and  availability  to  meet  ongoing  mission  requirements?  If  the  experience  loss  is 
restrained  to  a  specific  career  field,  do  the  effects  remain  localized  to  that  field,  or  are 
they  dispersed  across  the  unit? 
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Currently,  the  Air  Foree  relies  on  the  Logisties  Composite  Model  (LCOM)  to 
provide  an  answer  to  questions  surrounding  maintenance  manpower.  While  LCOM  is  an 
extremely  detailed  and  powerful  analytical  tool,  one  of  its  identified  shortcomings  is  its 
failure  to  address  issues  associated  with  individual  skill  levels  of  personnel,  to  include 
time  spent  in  training  junior  personnel  and  any  effects  due  to  differences  in  worker  skill 
levels.  This  research  focused  on  applying  a  new  modeling  methodology  to  the  sortie 
generation  scenario  while  specifically  capturing  a  variety  of  details  associated  with  the 
effects  of  individual  skill  levels.  A  key  feature  of  the  model  is  that  training  activities  are 
considered  which  can  cause  individual  maintenance  tasks  to  take  longer,  with  this  effect 
being  mitigated  over  time  as  each  individual  gains  experience.  The  concept  of  individual 
development  is  another  central  model  tenet;  each  maintainer  modeled  increases  their  job 
efficiency  over  time  according  to  a  skill-level  and  career-field-specific  learning  curve. 

The  prototype  model  developed  through  this  research  indicated  a  surprising 
amount  of  interconnectedness  between  disparate  career  fields  within  a  typical 
maintenance  unit.  Additionally,  there  was  an  indication  that  a  unit’s  capacity  for  sortie 
production  was  directly  tied  not  only  to  the  number  of  personnel  within  the  unit  but  also 
to  the  skill-  and  career-field  mix. 

With  a  constant  charge  to  maintain  capability  while  ensuring  an  overall  economy 
of  force,  leadership  must  be  able  to  make  informed  decisions  on  the  size  and  shape  of  our 
fighting  force.  End  strength  is  important,  but  this  raw  number  is  an  incomplete 
measurement.  Gaining  and  leveraging  an  understanding  of  the  range  of  effects  resulting 
from  these  underlying  personnel  interactions  is  critical  in  ensuring  we  remain  postured  as 
the  world’s  foremost  Air  Force. 
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APPENDIX  D.  Summary  Chart 
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